PlanB on Nostr: dave can you please make this Python code faster (data file d.txt is ~350GB and the ...
dave (npub1tsg…htk4) can you please make this Python code faster (data file d.txt is ~350GB and the code processes ~2 billion dictionary inserts and ~2 billion pops)?
d={}
with open('d.txt','r') as f:
for l in f:
v=l.split(" ")
b,i,o=int(v[0]),int(v[1]),int(v[2])
for x in range(4+i,4+i+o):
d[str(v[x+o])]=(b,float(v[x]))
for x in range(4,4+i):
d.pop(str(v[x]), None)
print(d)
Published at
2023-05-11 21:55:37Event JSON
{
"id": "b81f94cbf5617ade49958440872574102d3445c27850b8a35f5c52266cf62954",
"pubkey": "a3b0ce5d70d0db22885706b2b1f144c6864a7e4828acff3f8f01ca6b3f54ad15",
"created_at": 1683842137,
"kind": 1,
"tags": [
[
"p",
"5c10ed0678805156d39ef1ef6d46110fe1e7e590ae04986ccf48ba1299cb53e2"
],
[
"r",
"d.txt"
],
[
"r",
"d.pop"
]
],
"content": "nostr:npub1tsgw6pncspg4d5u778hk63s3pls70evs4czfsmx0fzap9xwt203qtkhtk4 can you please make this Python code faster (data file d.txt is ~350GB and the code processes ~2 billion dictionary inserts and ~2 billion pops)?\n\nd={}\nwith open('d.txt','r') as f:\n for l in f:\n v=l.split(\" \")\n b,i,o=int(v[0]),int(v[1]),int(v[2])\n for x in range(4+i,4+i+o): \n d[str(v[x+o])]=(b,float(v[x]))\n for x in range(4,4+i):\n d.pop(str(v[x]), None)\nprint(d)",
"sig": "dd9dbeac86b237f59b1e417e33c83b82aedd489583242ac4326bef37ff1bf60437b9d8fbc5757203c168ec49c7b0542f4698e8ebf770dcd22d6c74825fe89116"
}