python library for large tab/comma delimited text file -

-

i have big genomic data files analyze, come in 2 forms, 1 individual dosage file this:

id                      snp1    snp2    snp3    snp4    snp5    snp6 rs1->1000001    dose    1.994   1.998   1.998   1.998   1.830   1.335 rs1->1000002    dose    1.291   1.998   1.998   1.998   1.830   1.335 rs1->100001     dose    1.992   1.998   1.998   1.998   1.830   1.335 rs1->100002     dose    1.394   1.998   1.998   1.998   1.830   1.335 rs1->10001      dose    1.994   1.998   1.998   1.998   1.830   1.335 rs1->1001001    dose    1.904   1.998   1.998   1.998   1.830   1.335 rs1->1002001    dose    1.094   1.998   1.998   1.998   1.830   1.335 rs1->1003001    dose    1.994   1.998   1.998   1.998   1.830   1.335 rs1->1004001    dose    1.994   1.998   1.998   1.998   1.830   1.335 rs1->1005002    dose    1.994   1.998   1.998   1.998   1.830   1.335

the other contains summary info:

snp         al1 al2 freq1   maf     quality rsq  22_16050607 g     0.99699 0.00301 0.99699 0.00000 22_16050650 c   t   0.99900 0.00100 0.99900 0.00000 22_16051065 g     0.99900 0.00100 0.99900 0.00000 22_16051134   g   0.99900 0.00100 0.99900 0.00000 rs62224609  t   c   0.91483 0.08517 0.91483 -0.00000 rs62224610  g   c   0.66733 0.33267 0.66733 0.00000 22_16051477 c     0.99399 0.00601 0.99399 -0.00000 22_16051493 g     0.99900 0.00100 0.99900 -0.00000 22_16051497   g   0.64529 0.35471 0.64529 0.00000

the snp column in second file corresponds snp1, snp2... in first file. need use summary info in second file quality check , selection, apply statistical analysis on data in first file accordingly.

the question is, there python library suitable task? performance vital here, because these huge files. thanks!

for dealing large files , data high performance , efficient manipulation, there no better module pandas

the following code read file dataframe , allow easy manipulation:

import pandas pd data = 'my_data.csv' df = pd.read_csv(data)

now df efficient dataframe containing data! also, don't need it's tab delimiter because pandas "sniffs" delimiter


Comments

Post a Comment

Popular posts from this blog

java - Jmockit String final length method mocking Issue -

-
using jmockit 1.2 jar, trying mock string's length method getting unexpected invocation exception: failed: test java.lang.illegalstateexception: missing invocation mocked type @ point; please make sure such invocations appear after declaration of suitable mock field or parameter @ stringdemo.testa$1.<init>(testa.java:17) @ stringdemo.testa.test(testa.java:13) i using testng: @test public void test() throws exception { new expectations() { @mocked("length") string astring; { astring.length(); result = 2; } }; system.out.println(a.showa("test")); } } actual class a: public class { public static int showa(string str){ int a= str.length(); return a; } } this wrong way of recording expected results. shouldn't mock , record string's length() method, record showa() instead. here solution @suppresswarnings("unused...
Read more

What is the difference between data design and data model(ERD) -

-
i have google question , find answer in day. know, data design many many table have list column,and mark pk , fk. why er-diagram have find this? data design process of designing database. main output of data design detailed logical data model of database. some people data design includes of needed logical , physical design choices , physical storage parameters needed generate design in data definition language. technically, database analysis, database analysts (dbas) trained do. while person can both data design , database analysis, these 2 different tasks. data design models data. database analysis takes model , applies 1 or more database engines (relational, hierarchical, nosql). a logical data model 1 of main outputs of data design. data model represented entity relationship diagram, or er diagram.
Read more

ios - Can NSManagedObject conform to NSCoding -

-
i need transfer single object across device. right converting nsmanagedobject dictionary , archiving , sending nsdata. upon receiving unarchiving it. transfer nsmanagedobject archiving , unarchiving instead of creating intermediate data object. @interface test : nsmanagedobject<nscoding> @property (nonatomic, retain) nsstring * title; @end @implementation test @dynamic title; - (id)initwithcoder:(nscoder *)coder { self = [super init]; if (self) { self.title = [coder decodeobjectforkey:@"title"]; //<crash } return self; } - (void)encodewithcoder:(nscoder *)coder { [coder encodeobject:self.title forkey:@"title"]; } @end nsdata *archivedobjects = [nskeyedarchiver archiveddatawithrootobject:testobj]; nsdata *objectsdata = archivedobjects; if ([objectsdata length] > 0) { nsarray *objects = [nskeyedunarchiver unarchiveobjectwithdata:objectsdata]; } the problem above code is. crashes @ self.title in initwithcoder sa...
Read more