![]() You don’t need to master everything database related, but make sure when someone uses the word Acid you don’t say Yeah man, groovy. Have a skim through the top rated answer. It didn’t start off as a 5GB file, but grew over time till it became too large to open with Excel.ĭatabases routinely handle Gigabytes of data without any problems, so we should be using them. It works.įor an example, just look at the current file. Many scientists and engineers happy to use Ms Excel, CSV files, or even text files. You store the data in a database and read it. So how do you open a 5GB file? Not all of it at the same time is the answer. Not to mention, you may not even have enough memory to open it.Įven if you buy a more powerful machine, the lack of tools to work with large files will hold you back. Ms Excel can’t open it, Openoffice can’t open it, Notepad++ can’t open it. So the file we want to open is almost 5.3GB. Minor correction to video below: Sqlite can handle concurrent connections, though it may not be as fast. The videos are mixed with the transcripts, so scroll down if you are only interested in the videos. There are 6 videos + transcript in this series. Intro to SqLite.ipynb is the notebook we are working with. If you have used SQL with SqLite before, still have a look at the notebook, as I will be referring back to it. To start off, we will get a gentle intro to SQL. ![]() We will be using Ipython Notebooks( see here for a quick intro). While some people think this answer is common sense, I have seen far too many people intimidated with databases, so this series also serves as an introduction to SQL and Sqlite. The Problem: We have a 5GB file, and no tool we have can work with such large files (provided you even have enough free RAM to hold the whole thing in memory).
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |