python library for large tab/comma delimited text file -

-

i have big genomic data files analyze, come in 2 forms, 1 individual dosage file this:

id                      snp1    snp2    snp3    snp4    snp5    snp6 rs1->1000001    dose    1.994   1.998   1.998   1.998   1.830   1.335 rs1->1000002    dose    1.291   1.998   1.998   1.998   1.830   1.335 rs1->100001     dose    1.992   1.998   1.998   1.998   1.830   1.335 rs1->100002     dose    1.394   1.998   1.998   1.998   1.830   1.335 rs1->10001      dose    1.994   1.998   1.998   1.998   1.830   1.335 rs1->1001001    dose    1.904   1.998   1.998   1.998   1.830   1.335 rs1->1002001    dose    1.094   1.998   1.998   1.998   1.830   1.335 rs1->1003001    dose    1.994   1.998   1.998   1.998   1.830   1.335 rs1->1004001    dose    1.994   1.998   1.998   1.998   1.830   1.335 rs1->1005002    dose    1.994   1.998   1.998   1.998   1.830   1.335

the other contains summary info:

snp         al1 al2 freq1   maf     quality rsq  22_16050607 g     0.99699 0.00301 0.99699 0.00000 22_16050650 c   t   0.99900 0.00100 0.99900 0.00000 22_16051065 g     0.99900 0.00100 0.99900 0.00000 22_16051134   g   0.99900 0.00100 0.99900 0.00000 rs62224609  t   c   0.91483 0.08517 0.91483 -0.00000 rs62224610  g   c   0.66733 0.33267 0.66733 0.00000 22_16051477 c     0.99399 0.00601 0.99399 -0.00000 22_16051493 g     0.99900 0.00100 0.99900 -0.00000 22_16051497   g   0.64529 0.35471 0.64529 0.00000

the snp column in second file corresponds snp1, snp2... in first file. need use summary info in second file quality check , selection, apply statistical analysis on data in first file accordingly.

the question is, there python library suitable task? performance vital here, because these huge files. thanks!

for dealing large files , data high performance , efficient manipulation, there no better module pandas

the following code read file dataframe , allow easy manipulation:

import pandas pd data = 'my_data.csv' df = pd.read_csv(data)

now df efficient dataframe containing data! also, don't need it's tab delimiter because pandas "sniffs" delimiter


Comments

Post a Comment

Popular posts from this blog

java - Jmockit String final length method mocking Issue -

-
using jmockit 1.2 jar, trying mock string's length method getting unexpected invocation exception: failed: test java.lang.illegalstateexception: missing invocation mocked type @ point; please make sure such invocations appear after declaration of suitable mock field or parameter @ stringdemo.testa$1.<init>(testa.java:17) @ stringdemo.testa.test(testa.java:13) i using testng: @test public void test() throws exception { new expectations() { @mocked("length") string astring; { astring.length(); result = 2; } }; system.out.println(a.showa("test")); } } actual class a: public class { public static int showa(string str){ int a= str.length(); return a; } } this wrong way of recording expected results. shouldn't mock , record string's length() method, record showa() instead. here solution @suppresswarnings("unused&qu
Read more

asp.net - Razor Page Hosted on IIS 6 Fails Every Morning -

-
i have simple page in intranet uses razor/asp fetch single record table , display it, plus few graphic, dashboard display. meaning, no user intervention other first time open ie internal url. the problem having every morning ie displays "page not found or network error" message. refresh page, , it's stuck , doesn't display anything. try different pc, open ie internal url , it's stuck... ...until following: login server hosting page run inetmgr go web sites, etc right click on page that's giving me problem , select browse at moment, error message: server error in '/' application. -------------------------------------------------------------------------------- type of page not served. description: type of page have requested not served because has been explicitly forbidden. extension '.cshtml' may incorrect. please review url below , make sure spelled correctly. requested url: /application/dashboard.cshtml ---------------
Read more

c++ - wxwidget compiling on windows command prompt -

-
can please guide me on method,as how compile wxwidget using windows command prompt or msys. [i can't seem find on wxwiki/wxwidget official book/anywhere else] i have compiled wxwidget instructions have provided (using msys). from searching on internet seems 1 can (maybe?) through use of makefile (is correct?) if yes: directory & how should link wx libraries in makefile if no: way besides makefile? i know can use ide's code::block , make life simpler prefer compile using command prompt/msys. thanks in advance.
Read more