When grep isn't good enough, try Blackbird
GitHub has a lot of code to search – more than 200 million repositories – and says last November's beta version of a search engine optimized for source code that has caused a"flurry of innovation."
GitHub engineer Timothy Clem explained that the company has had problems getting existing technology to work well."The truth is from Solr to Elasticsearch, we haven't had a lot of luck using general text search products to power code search," he in a GitHub Universe video presentation."The user experience is poor. It's very, very expensive to host and it's slow to index."on Monday, Clem delved into the technology used to scour just a quarter of those repos, a code search engine built in Rust called Blackbird.
Blackbird currently provides access to almost 45 million GitHub repositories, which together amount to 115TB of code and 15.5 billion documents. Shifting through that many lines of code requires something stronger than grep, a common command line tool on Unix-like systems for searching through text data.on an 8-core Intel CPU to run an exhaustive regular expression query on a 13GB file in memory, Clem explained, takes about 2.769 seconds, or 0.6GB/sec/core.
"We can see pretty quickly that this really isn’t going to work for the larger amount of data we have," he said."Code search runs on 64 core, 32 machine clusters. Even if we managed to put 115TB of code in memory and assume we can perfectly parallelize the work, we’re going to saturate 2,048 CPU cores for 96 seconds to serve a single query! Only that one query can run. Everybody else has to get in line.
Österreich Neuesten Nachrichten, Österreich Schlagzeilen
Similar News:Sie können auch ähnliche Nachrichten wie diese lesen, die wir aus anderen Nachrichtenquellen gesammelt haben.
GitHub CEO: EU AI Act shouldn't apply to open source devsLawmakers said to be trying to align on the basics by 'early March'
Weiterlesen »
Evri couriers claim that they are 'treated like slaves' at work“By the time you’ve sorted the parcels you’ve been there for two hours without getting paid, getting absolutely soaked if it’s raining or standing in the boiling sunshine.'
Weiterlesen »
New State Pension payment delay warning for people reaching retirement ageThis regular payment is available for those who have reached the UK Government’s eligible retirement age, which is currently 66 for both men and women
Weiterlesen »
February’s PlayStation Plus games are now available to claim | VGCSony has begun rolling out February's PlayStation Plus Essential games. They are OlliOlli World, Mafia Definitive Edition, Evil Dead: The Game and Destiny 2: Beyond Light
Weiterlesen »
Eurovision: Volunteer roles look and smell like a job, unions claimThe unpaid roles advertised by Liverpool City Council 'look like and smell like a job', unions say.
Weiterlesen »