One of my favorite moments in HN history was watching the authors of the various search tools decide on a common ".ignore" file as opposed to each having their own: https://news.ycombinator.com/item?id=12568245
I would argue that grep-like tools which read .gitignore violate the Principle of Least Astonishment (POLA). It would be fine if there were a --ignore flag to enable such functionality, but defaulting to it just feels wrong to me. Obviously smarter people than I disagree, but my dumdum head just feels that way.
Back in the day I would have agreed with you, but ever since there is js everywhere you end up with minified js that megabytes big and match everything. I still have muscle memory with `grep -r` and it almost always ends up with some js file, that I didn't know exists ruining the moment.
> Obviously smarter people than I disagree, but my dumdum head just feels that way.
That's absolutely not it. What you're describing is part of the UNIX philosophy: programs should do one thing and do it well, and they should function in a way that makes them very versatile and composable, etc.
And that part of the philosophy works GREAT when everything follows another part of the philosophy: everything should be based on flat text files.
But for a number of reasons, and regardless of whatever we all think of those reasons, we live in a world that has a lot of stuff that is NOT the kind of flat text file grep was made for. Binary formats, minified JS, etc. And so to make the tool more practical on a modern *nix workstation, suddenly more people want defaults that are going to work on their flat text files and transparently ignore things like .git.
It's just that you've showed up to an wildly unprincipled world armed with principles.
I have to agree here. I love ripgrep, but at times I've had to go back to regular grep because I couldn't figure out what it was ignoring and why, and there were far too many settings to figure it out.
It's good if they can share syntax. You use the same English words to ask Alice and Bob questions, but when you say "So, tell me, Alice…" you don't want Bob to answer you instead. Using another tool's config by default, making it difficult/impossible to use the dedicated config is the most annoying thing I can imagine. If that's what rg does, I guess that must be the reason I couldn't switch.
I’ve read this multiple times over the years and this post is still the most interesting and informative piece describing the problem of making a fast grep-like tool. I love that it doesn’t just describe how ripgrep works but also how all the other tools work and then compares the various techniques. It’s simultaneously a tutorial and an expert deep dive. Just a beautiful piece of writing. In a perfect world, all code would be similarly documented.
Such a good read. I actually went back though it the other day to steal the searching for the least common byte idea out to speed up my search tool https://github.com/boyter/cs which when coupled with the simd upper lower search technique from fzf cut the wall clock runtime by a third.
There was this post from cursor https://cursor.com/blog/fast-regex-search today about building an index for agents due to them hitting a limit on ripgrep, but I’m not sure what codebase they are hitting that warrants it. Especially since they would have to be at 100-200 GB to be getting to 15s of runtime. Unless it’s all matches that is.
Yeah, that Cursor blog post is a bit iffy since they just brush over the "ripgrep is slow on large monorepos", move on to techniques they used, and then completely ignore the fact that you have to build and maintain the index.
On a mid-size codebase, I fzf- and rg-ed through the code almost instantly, while watching my coworker's computer slow down to a crawl when Pycharm started reindexing the project.
Is IRIX experiencing a hobbyist revival or something? This is the second IRIX reference I’ve seen on here in the past two days, and there was a submission a day or two ago (c.f. a Voodoo video card?) as well. I haven’t personally encountered IRIX in the wild since a company I worked at in 2003. I suppose SGI has always had a cool factor but it’s unusual seeing it come up in a cluster of mentions like this.
SGUG tried hard to port newer packages for IRIX for several years but hit a wall with ABI mismatches leading to GOT corruption. This prevented a lot of larger packages from working or even building.
I picked up the effort again after wondering if LLMs would help. I ran into the ABI problems pretty quickly. This time though, I had Claude use Ghidra to RE the IRIX runtime linker daemon, which gave the LLM enough to understand that the memory structures I’d been using in LLVM were all wrong. See https://github.com/unxmaal/mogrix/blob/main/rules/methods/ir... .
After cracking that mystery I was able to quickly get “impossible” packages building, like WebKit, QT5, and even small bits of Go and Rust.
I’m optimistic that we’ll see more useful applications built for this cool old OS.
That is pretty neat. I guess this sort of unlocking and unblocking effort is exactly what’s needed for a revival.
I’m sort of thinking of AmigaOS/Workbench as well although, perhaps because of what I would assume was always a much larger user base than SGI had, it maybe never went away like SGI and IRIX did.
It is great seeing these old platforms get a new lease of life.
Ooh that's super interesting. I assume you shared the recipe with the irix community? I remember keeping Netscape up to date on my Indy was already a struggle in 2002.
When I first heard about ripgrep my reaction was laughing. grep had been too established. No way something that isn't 100% compatible with grep could get any traction.
And I was dead wrong. Overnight everyone uses rg (me included).
I don't remember why I didn't switch from ag, but I remember it was a conscious decision. I think it had something to do with configuration, rg using implicit '.ignore' file (a super-generic name instead of a proper tool-specific config) or even .gitignore, or something else very much unwarranted, that made it annoying to use. Cannot remember, really, only remember that I spent too much time trying to make it behave and decided it isn't worth it. Anyway, faster is nice, but somehow I don't ever feel that ag is too slow for anything. The switch from the previous one (what was it? ack?) felt like a drastic improvement, but ag vs. rg wasn't much difference to me in practice.
Totally agree with the author of rg here. Config names should be unambiguous. Anyway, must have been something else, then. As I've said, I cannot remember what was the specific problem, only that it wasn't quite compatible with the workflow I was used to, and now it'd take another full-in attempt to switch to figure out what was so annoying to me back then.
I was just trying to remember why I switched _to_ rg.
It's nice and everything, but I remember being happy with the tools before (I think i moved from grep to ack, then jumped due to perf to ag and for unremembered reasons to pt.)
It took me a while, but I remembered I ran into an issue with pt incorrectly guessing the encoding of some files[0].
I can't remember whether rg suffered from the same issue or not, but I do know after switching to rg everything was plain sailing and I've been happy with it since.
For me it was trying to add a filter to search CMake files to ag and then realizing that the code had some rather stupid design decisions that prevented it. I wrote a pull request that fixed enough things to add the filter, got ignored by the maintainer and later realized that other people had already written the same filter and were ignored too.
I was using ripgrep once and it had a bug that led me downa terrifying rabbit hole - I can't recall what it was but it involved not being able to find text that absolutely should have been there.
Eventually I was considering rebuilding the machine completely but for some reason after a very long time digging deep into the rabbit hole I tried plain old grep and there was the data exactly where it should have been.
So it's such a vague story but it was a while back - I don't remember the specifics but I sure recall the panic.
Sometimes I forget that some of the config files I have for CI in a project are under a dot directory, and therefore ignored by rg by default, so I have to repeat the search giving the path to that config files subdirectory if I want to see the results that are under that one (or use some extra flags for rg to not ignore dot directories other than .git)
Was the file in a .gitignore by any chance? I've got my home folder in git to keep track of dot/config files and that always catches me out. Really dislike it defaulting to that ignoring files that are ignored by git.
I had that happen too recently… Basically rg x would show nothing but grep -r x showed the lines for any x. Tried multiple times with different x, then I kept using grep -r at that time. After a few days, I started using rg again and it worked fine but now I tend to use grep -r occasionally too to make sure.
Next time that happens try looking at the paths, adding a pair of -u, or running with --debug: by default rg will ignore files which are hidden (dotfiles) or excluded by ignore files (.gitignore, .ignore, …).
I use "grep" to search files (it should never skip any unless I tell it to do otherwise) and "git grep" to be a programmer searching a codebase (where it should only look at code files unless I tell it to do otherwise). Two different hats.
I wouldn't want to use tools that straddle the two, unless they had a nice clear way of picking one or the other. ripgrep does have "--no-ignore", though I would prefer -a / --all (one could make their own with alias rga='rg --no-ignore')
And burntsushi is one of us: he's regularly here on HN. Big thanks to him. As soon as rg came out I was building it on Linux. Now it ships stocks with Debian (since Bookworm? Don't remember): thanks, thanks and more thanks.
> > IMO, as long as the time differences remain small, I'm totally okay with ripgrep being slower by default on smaller corpora if it means being a lot faster by default on bigger corpora.
Note that this is the author of ripgrep replying to a third party commenter asking whether rg isn’t already lightweight, and comparing the two under various possible definitions of “lightweight”.
I don’t understand when people typeset some name in verbatim, lowercase, but then have another name for the actual command. That’s confusing to me.
Programmers are too enarmored with lower-case names. Why not Ripgrep? Then I can surmise that there might not be some program ripgrep(1) (there might be a shorter version), since using capital letters is not traditional for CLI programs.
> Stacked Git, StGit for short, is an application for managing Git commits as a stack of patches.
> ... The `stg` command line tool ...
Now, I’ve been puzzled in the past when inputing `stgit` doesn’t work. But here they call it StGit for short and the actual command is typeset in verbatim (stg(1) would have also worked).
Because we are constantly writing variables that are lowercase. Coming up with a name that is both short but immediately understandable is what we live for. Variables are our shrine, we stare at them everyday and are used to their beauty and simplicity.
How would you capitalise it? RipGrep? RIPGrep? You’d need to pick a side and lose the pun. (And of course grep itself would need to be GReP if we took it all the way)
The fun part is it is pretty easy to “rewrite” ripgrep in rust, because burntsushi wrote it as a ton of crates which you can reuse. So you can reuse this to build your own with blackjack and hookers.
A "ton of crates" is IMO the best way to write large Rust programs. Each crate in Rust is a compilation unit, the equivalent of one `.c` file in C. If they don't depend on one another, each crate can be compiled in parallel. It makes building the whole project faster, often significantly so. As with anything one can take it too far, but as long as each crate makes sense as an independent unit it's good.
Gotta add a +1 for this. I wanted to do some ignore files etc for a project.
I thought "well I kinda want to do what rg does". Had a little glance and it was already nicely extracted into a separate crate that was a dream to use.
It seems to me that `rg` is the number one most important part that enables LLMs to be smart agents in a codebase. Who would have thought that a code search tool would enable AGI?
Faster is not always the best thing. I still remember when vs code changed to ripgrep I had to change my habit using it, before then I can just open vs code to any folder and do something with it, even if the folder contains millions of small text files. It worked fine before, but then rg was picked, and it happily used all of my cpu cores scanning files, made me unable to do anything for awhile.
To be honest I hate all the new rust replacement tools, they introduce new behavior just for the sake of it, it's annoying.
That's absolutely not it. What you're describing is part of the UNIX philosophy: programs should do one thing and do it well, and they should function in a way that makes them very versatile and composable, etc.
And that part of the philosophy works GREAT when everything follows another part of the philosophy: everything should be based on flat text files.
But for a number of reasons, and regardless of whatever we all think of those reasons, we live in a world that has a lot of stuff that is NOT the kind of flat text file grep was made for. Binary formats, minified JS, etc. And so to make the tool more practical on a modern *nix workstation, suddenly more people want defaults that are going to work on their flat text files and transparently ignore things like .git.
It's just that you've showed up to an wildly unprincipled world armed with principles.
--ignore-file=.ignore
--ignore-file=.gitignore
--ignore-file=.dockerignore
--ignore-file=.npmignore
etc
but then, assuming all those share the same "ignore file syntax/grammar"...
But that’s the kind of problem that only successful things have to worry about.
There was this post from cursor https://cursor.com/blog/fast-regex-search today about building an index for agents due to them hitting a limit on ripgrep, but I’m not sure what codebase they are hitting that warrants it. Especially since they would have to be at 100-200 GB to be getting to 15s of runtime. Unless it’s all matches that is.
On a mid-size codebase, I fzf- and rg-ed through the code almost instantly, while watching my coworker's computer slow down to a crawl when Pycharm started reindexing the project.
It’s fast even on a 300mhz Octane.
SGUG tried hard to port newer packages for IRIX for several years but hit a wall with ABI mismatches leading to GOT corruption. This prevented a lot of larger packages from working or even building.
I picked up the effort again after wondering if LLMs would help. I ran into the ABI problems pretty quickly. This time though, I had Claude use Ghidra to RE the IRIX runtime linker daemon, which gave the LLM enough to understand that the memory structures I’d been using in LLVM were all wrong. See https://github.com/unxmaal/mogrix/blob/main/rules/methods/ir... .
After cracking that mystery I was able to quickly get “impossible” packages building, like WebKit, QT5, and even small bits of Go and Rust.
I’m optimistic that we’ll see more useful applications built for this cool old OS.
I’m sort of thinking of AmigaOS/Workbench as well although, perhaps because of what I would assume was always a much larger user base than SGI had, it maybe never went away like SGI and IRIX did.
It is great seeing these old platforms get a new lease of life.
And I was dead wrong. Overnight everyone uses rg (me included).
The ".ignore" name was actually suggested by the author of ag (whereas the author of rg thought it was too generic): https://news.ycombinator.com/item?id=12568245
It's nice and everything, but I remember being happy with the tools before (I think i moved from grep to ack, then jumped due to perf to ag and for unremembered reasons to pt.)
It took me a while, but I remembered I ran into an issue with pt incorrectly guessing the encoding of some files[0].
I can't remember whether rg suffered from the same issue or not, but I do know after switching to rg everything was plain sailing and I've been happy with it since.
[0] https://github.com/monochromegane/the_platinum_searcher/issu...
Eventually I was considering rebuilding the machine completely but for some reason after a very long time digging deep into the rabbit hole I tried plain old grep and there was the data exactly where it should have been.
So it's such a vague story but it was a while back - I don't remember the specifics but I sure recall the panic.
Sometimes I forget that some of the config files I have for CI in a project are under a dot directory, and therefore ignored by rg by default, so I have to repeat the search giving the path to that config files subdirectory if I want to see the results that are under that one (or use some extra flags for rg to not ignore dot directories other than .git)
I still use it but Ive never trusted it fully since then I double check.
It's the reason I started using it. Got sick of grep returning results from node_modules etc.
See https://github.com/BurntSushi/ripgrep/blob/master/GUIDE.md#a... for the details.
I wouldn't want to use tools that straddle the two, unless they had a nice clear way of picking one or the other. ripgrep does have "--no-ignore", though I would prefer -a / --all (one could make their own with alias rga='rg --no-ignore')
I think riggrep will not search UTF-16 files by default. I had some such issue once at least.
I ran into that with pt, and it definitely made me think I was going mad[0]. I can't fully remember if rg suffered from the same issue or not.
[0] https://github.com/monochromegane/the_platinum_searcher/issu...
Someone please make an awesome new sed and awk.
https://github.com/chmln/sd
// really hoping openai wouldn't now force him to work on some crappy codex stuff if he stays there / in astral.
https://reddit.com/r/rust/comments/1fvzfnb/gg_a_fast_more_li...
Also something-something about dependencies (a Rust staple): https://www.reddit.com/r/rust/comments/1fvzfnb/gg_a_fast_mor...
The TUI is great, and approximate matches are insanely useful.
I don’t understand when people typeset some name in verbatim, lowercase, but then have another name for the actual command. That’s confusing to me.
Programmers are too enarmored with lower-case names. Why not Ripgrep? Then I can surmise that there might not be some program ripgrep(1) (there might be a shorter version), since using capital letters is not traditional for CLI programs.
Look at Stacked Git:
https://stacked-git.github.io/
> Stacked Git, StGit for short, is an application for managing Git commits as a stack of patches.
> ... The `stg` command line tool ...
Now, I’ve been puzzled in the past when inputing `stgit` doesn’t work. But here they call it StGit for short and the actual command is typeset in verbatim (stg(1) would have also worked).
You may be able to download ripgrep, and execute it (!), but god forbid you can create an alias in your shell in a persistant manner.
Really? "most" even? What CAN you do if you can't edit files in your own $HOME?
I thought "well I kinda want to do what rg does". Had a little glance and it was already nicely extracted into a separate crate that was a dream to use.
Thanks @BurntSushi!
Someone kinda did
To be honest I hate all the new rust replacement tools, they introduce new behavior just for the sake of it, it's annoying.