LinHES Forums
http://forums.linhes.org/

md5sum, corrupt download? and bittorrent.
http://forums.linhes.org/viewtopic.php?f=1&t=10399		 Page 1 of 1
Author:  Ramrunner [ Wed Jun 07, 2006 6:04 pm ]
Post subject:  md5sum, corrupt download? and bittorrent.
Hey there, I've been running R5B7 successfully for a long time, except for not being able to get mythburn to work. I saw that R5C7 had updated scripts to do with that so I thought I'd download it and give that a try.

Now before I mention this and get "just download it again" please be aware I only have dial-up capabilities and it takes about 30 or so hours to get an ISO.

I've downloaded the iso from the listed Australian mirror, but the md5sum does not match.

I've tried to find rsync servers to correct my download, but none seem to exist.

So, I finally found a torrent for the iso, and figured I'd just use that to correct my download.

The problem is the iso seems to hash just fine and tells me I've downloaded the file 100%.

I've tried checking the md5 in a windows GUI program, Fedora, in both binary and ACSII modes (just to check my sanity) but none match the 38970f66ec2c8e29467da02c70dae55f listed. I should mention that my KnoppMythR5B7.iso file matched the md5sum the first time and was downloaded the same way.

Please, someone save my sanity and tell me what I can do to fix this download.....

Regards,
Ramrunner
Author:  tjc [ Wed Jun 07, 2006 6:44 pm ]
Post subject: 
Sorry, there's just no way of knowing what part of the file is corrupt. Unpleasant as it is it's time for a do over.

Of course you could also explore alternatives like making friends with one of the closer KM users who has a high speed connection and get them to mail you a working CD.
Author:  Ramrunner [ Wed Jun 07, 2006 7:25 pm ]
Post subject: 
No way. The amount of things I had to put on hold just to get the ISO far outweigh it's benefits.

When you say there's no way to confirm which part of the file is corrupt, that is fortunately incorrect.

rsync and bittorrent BOTH will perform checksums on the file and "fill in the blanks" if the file is corrupt. There ARE ways to fix this file.

I'll change my original question:

1> Does anyone have an rsync server hosting R5C7 I can check with?
2> If my torrent client (uTorrent) says the file is 100% complete and drops it in my "completed" folder, then basically, torrent is saying the file is 100% correct. The torrent checksum matches the file right? There's nothing I'm missing right?
3> Can someone confirm the listed checksum on the download page is correct?
4> If I'm still going insane, anyone who would like to send a "checksum matched" CD to Oakford (Perth), Western Australia (for a fee if you wish), could you please reply to this post.

Regards,
Ramrunner
[/quote]
Author:  tjc [ Wed Jun 07, 2006 8:05 pm ]
Post subject: 
Ramrunner wrote:
When you say there's no way to confirm which part of the file is corrupt, that is fortunately incorrect.

rsync and bittorrent BOTH will perform checksums on the file and "fill in the blanks" if the file is corrupt. There ARE ways to fix this file.

Didn't you try that already? At least with bit torrent? I did suggest using physical media for a reason. (I think the appropriate Classic CS quote is "Never under estimate the bandwith of a station wagon full of mag tape".)

Yes, a different partial updater with a different chunking or different checksuming method for those chunks might spot the corrupt section or sections. But only if the chunk sizes are different and/or the checksum algorithm used is different and even at that this is a probabilistic game. This all presumes that there wasn't some kind of short segment problem (a pretty good bet here) which shifted everything after point X. The reason for doing an MD5 or better still an SHA1 checksum at the end is that they have enough 9's (99.9...% chance of spotting ...) to deal with this much data.

Hmmm, that give me an evil idea... Hang on and I'll publish some chunkwise MD5 lists which will let you do a binary search down to the 1-2 hour range.
Author:  tjc [ Wed Jun 07, 2006 8:47 pm ]
Post subject: 
Here you go. Split your ISO into 16Mb chunks and let me know which ones don't match, then I'll split those down to half Mb chunks.
Code:
[tjc@silver KnoppMythR5C7.chunks]$ split -d -b 16777216 ../KnoppMythR5C7.iso KnoppMythR5C7.iso.
[tjc@silver KnoppMythR5C7.chunks]$ md5sum KnoppMythR5C7.iso.*
27c3b1e4b50a6c31a40ac87a29aeaa75  KnoppMythR5C7.iso.00
5ff9a3de46267567343726f577011021  KnoppMythR5C7.iso.01
03a4e5171f69fc02e608fd2c685c123e  KnoppMythR5C7.iso.02
d7343f6f5c56f7b9355cdce9e0a9850a  KnoppMythR5C7.iso.03
09a9571132542b14078a48816bccd46a  KnoppMythR5C7.iso.04
34231a422ccdbbae71fa1732b09e4c79  KnoppMythR5C7.iso.05
c8f28c4deb2469fb8628c60f4a1674aa  KnoppMythR5C7.iso.06
fe21605c13a1143724c10932f876463a  KnoppMythR5C7.iso.07
d953a5c8b6837fb51601ebbeeca93222  KnoppMythR5C7.iso.08
ec8e33c842a472d059bc2bfbdb00ea81  KnoppMythR5C7.iso.09
2908c305a96b8d14208746128d41c5c5  KnoppMythR5C7.iso.10
290d3d2496d0fee1e004290f616eb9df  KnoppMythR5C7.iso.11
9b977067972225b7703189cdd038ebc0  KnoppMythR5C7.iso.12
36776b93021352a696fccb273a7ea0d9  KnoppMythR5C7.iso.13
bb8c789b174c5c632c6a12bdcf8201a4  KnoppMythR5C7.iso.14
75c7b1cf219ab12328db5f40f9d03ff0  KnoppMythR5C7.iso.15
36efcdfcc900af847f3fe41fa2636e19  KnoppMythR5C7.iso.16
bc97b5c02f0cfb1f34c5282c448e310d  KnoppMythR5C7.iso.17
812cc6452bfd7d1d51be6a4f4e32d073  KnoppMythR5C7.iso.18
1d7822e6bb04e025878e4a417f520ab2  KnoppMythR5C7.iso.19
050130e857e07cebf497b9f30bcfc5f9  KnoppMythR5C7.iso.20
78e7c05cb0656f875f688cba8cc4a98a  KnoppMythR5C7.iso.21
49686b2b2cd527362b0117c0aca16c21  KnoppMythR5C7.iso.22
27240b16d49552c677ac2904b143466d  KnoppMythR5C7.iso.23
ef122000fb00abe90c8c0e178cf7eb7d  KnoppMythR5C7.iso.24
59c155c431e4ee0aeefa0f1c74773be3  KnoppMythR5C7.iso.25
f834f597cd1dadedb6303bb11733aab6  KnoppMythR5C7.iso.26
90a06b1e2ce8f1f35c8c72de84a53efb  KnoppMythR5C7.iso.27
37333ca971a2d427832dd5304c6390e1  KnoppMythR5C7.iso.28
5fc8b7a05bd876e98588880c690bed4a  KnoppMythR5C7.iso.29
2e6939bfe009c3dab539c07864ab20f9  KnoppMythR5C7.iso.30
44d4d4ed8cd99cc024f0907efbc46fc1  KnoppMythR5C7.iso.31
12b5698ec5e3104444fd9a0a7ff4f9e6  KnoppMythR5C7.iso.32
e950a9907bc2affcb6991f9f0fab290c  KnoppMythR5C7.iso.33
9b8a277ffe1fc18d705533db88d96709  KnoppMythR5C7.iso.34
e56336edab7802206a6b389baab063e6  KnoppMythR5C7.iso.35
ac178d03aa2999530e156af00d6aab3f  KnoppMythR5C7.iso.36
cb04a3aabdc2e0a37778b3a6b5b570b4  KnoppMythR5C7.iso.37
03e44133734070a1db8fd3650ee3564c  KnoppMythR5C7.iso.38
a3112e7bbf55da93522e389e2d980082  KnoppMythR5C7.iso.39
ec342cff0f1ecae1ade1471a2ce5e8d8  KnoppMythR5C7.iso.40
32ab737c99d52c0fcd81a6d5668fc752  KnoppMythR5C7.iso.41
8903b476c703136f56664982b0c50a84  KnoppMythR5C7.iso.42
Author:  Ramrunner [ Wed Jun 07, 2006 9:02 pm ]
Post subject: 
tjc - you're a devilishly evil man - I like it.

I just got called away on a job and HAVE to leave now.

However, as soon as I get back I'll do the split as per your instructions and post the results.

Sorry to do this to you, but I need the money, it's my job.

I promise as soon as I get back (this afternoon my time), I'll continue.

Ed.
Author:  Ramrunner [ Thu Jun 08, 2006 5:26 am ]
Post subject: 
Here's mine:

27c3b1e4b50a6c31a40ac87a29aeaa75 KnoppMythR5C7.iso.00
5ff9a3de46267567343726f577011021 KnoppMythR5C7.iso.01
03a4e5171f69fc02e608fd2c685c123e KnoppMythR5C7.iso.02
d7343f6f5c56f7b9355cdce9e0a9850a KnoppMythR5C7.iso.03
09a9571132542b14078a48816bccd46a KnoppMythR5C7.iso.04
34231a422ccdbbae71fa1732b09e4c79 KnoppMythR5C7.iso.05
c8f28c4deb2469fb8628c60f4a1674aa KnoppMythR5C7.iso.06
fe21605c13a1143724c10932f876463a KnoppMythR5C7.iso.07
d953a5c8b6837fb51601ebbeeca93222 KnoppMythR5C7.iso.08
ec8e33c842a472d059bc2bfbdb00ea81 KnoppMythR5C7.iso.09
2908c305a96b8d14208746128d41c5c5 KnoppMythR5C7.iso.10
290d3d2496d0fee1e004290f616eb9df KnoppMythR5C7.iso.11
9b977067972225b7703189cdd038ebc0 KnoppMythR5C7.iso.12
36776b93021352a696fccb273a7ea0d9 KnoppMythR5C7.iso.13
bb8c789b174c5c632c6a12bdcf8201a4 KnoppMythR5C7.iso.14
75c7b1cf219ab12328db5f40f9d03ff0 KnoppMythR5C7.iso.15
36efcdfcc900af847f3fe41fa2636e19 KnoppMythR5C7.iso.16
bc97b5c02f0cfb1f34c5282c448e310d KnoppMythR5C7.iso.17
812cc6452bfd7d1d51be6a4f4e32d073 KnoppMythR5C7.iso.18
1d7822e6bb04e025878e4a417f520ab2 KnoppMythR5C7.iso.19
050130e857e07cebf497b9f30bcfc5f9 KnoppMythR5C7.iso.20
78e7c05cb0656f875f688cba8cc4a98a KnoppMythR5C7.iso.21
49686b2b2cd527362b0117c0aca16c21 KnoppMythR5C7.iso.22
27240b16d49552c677ac2904b143466d KnoppMythR5C7.iso.23
ef122000fb00abe90c8c0e178cf7eb7d KnoppMythR5C7.iso.24
59c155c431e4ee0aeefa0f1c74773be3 KnoppMythR5C7.iso.25
f834f597cd1dadedb6303bb11733aab6 KnoppMythR5C7.iso.26
90a06b1e2ce8f1f35c8c72de84a53efb KnoppMythR5C7.iso.27
37333ca971a2d427832dd5304c6390e1 KnoppMythR5C7.iso.28
5fc8b7a05bd876e98588880c690bed4a KnoppMythR5C7.iso.29
2e6939bfe009c3dab539c07864ab20f9 KnoppMythR5C7.iso.30
44d4d4ed8cd99cc024f0907efbc46fc1 KnoppMythR5C7.iso.31
12b5698ec5e3104444fd9a0a7ff4f9e6 KnoppMythR5C7.iso.32
***(d13d85ae7111028b25923eefee6d0396 KnoppMythR5C7.iso.33)***
9b8a277ffe1fc18d705533db88d96709 KnoppMythR5C7.iso.34
e56336edab7802206a6b389baab063e6 KnoppMythR5C7.iso.35
ac178d03aa2999530e156af00d6aab3f KnoppMythR5C7.iso.36
cb04a3aabdc2e0a37778b3a6b5b570b4 KnoppMythR5C7.iso.37
03e44133734070a1db8fd3650ee3564c KnoppMythR5C7.iso.38
a3112e7bbf55da93522e389e2d980082 KnoppMythR5C7.iso.39
ec342cff0f1ecae1ade1471a2ce5e8d8 KnoppMythR5C7.iso.40
32ab737c99d52c0fcd81a6d5668fc752 KnoppMythR5C7.iso.41
8903b476c703136f56664982b0c50a84 KnoppMythR5C7.iso.42

As you can see, chunk 33 is stuffed (I've marked it). BitTorrent did not fix it.

When I had this in the past (only twice - with Fedora DVD downloads), rsync fixed it.

I guess BitTorrent doesn't (this is the first time Ive tried BitTorrent for fixing downloads.

I guess unless there is an rsync server that I don't know about, someone can post me the CD, or you'd be kind enough to post or split chunk 33 down further I'm still screwed.

Thanks for your help so far. At least I've learned BitTorrent, contrary to what is posted on the internet, cannot fix a broken download.

I would have thought it does a complete hash check on each chunk (like rsync does) but obviously it doesn't.

Regards,
Ed.
Author:  tjc [ Thu Jun 08, 2006 7:48 pm ]
Post subject: 
Well here's one simple possibility, truncate the file at the last known good chunk and resume from there. In this case that means you're 75% of the way there. A 7.5 hour DL would still suck, but only 25% as much...

I don't remember enough about the details of bit torrent to know if it's possible but if the "neded chunk list" is an external file, the possibility of hacking it exists. Just generate one that lists the good chinks as "have" and bad one as "don't have". We could also play divide and conquer a bit more and possibly get the chunk size way down to something manageable.

Code:
[tjc@silver KnoppMythR5C7.chunks]$ split -d -b 524288 KnoppMythR5C7.iso.33 KnoppMythR5C7.iso.33.
[tjc@silver KnoppMythR5C7.chunks]$ md5sum KnoppMythR5C7.iso.33.*
e66d0366752da48ab686f5a5d9fbf498  KnoppMythR5C7.iso.33.00
4ea9cd69e5341373bb0ec0bb0ca3c3db  KnoppMythR5C7.iso.33.01
25f6336115f6b6e3fa5bc1d43116887e  KnoppMythR5C7.iso.33.02
2c43be4715b5a9ecc9e29a34511c0b35  KnoppMythR5C7.iso.33.03
002eb6b8f0807ef9524695d7cfb5de19  KnoppMythR5C7.iso.33.04
33e323ecb926e1492cf78b056ce3c099  KnoppMythR5C7.iso.33.05
19c9ec9f19de64d45599974d2e248eac  KnoppMythR5C7.iso.33.06
b37af4bfa58f801ad55816e500fa3dad  KnoppMythR5C7.iso.33.07
bcf4630de00ab845e103e4c36d96e80d  KnoppMythR5C7.iso.33.08
c657e4d306700393a4e68b5af6efdea4  KnoppMythR5C7.iso.33.09
e48966d03eb0dba101ecffa740f61fc0  KnoppMythR5C7.iso.33.10
1b76bc21c2722c3494c7cd9262ec2791  KnoppMythR5C7.iso.33.11
6d7f91c1432155be95292a4b42519129  KnoppMythR5C7.iso.33.12
cc1869b5278050ea9497a896ce76cd52  KnoppMythR5C7.iso.33.13
f1d16194397750438cb6952f59d157a2  KnoppMythR5C7.iso.33.14
a9a348292e8ef71fcdc12a03898b8ccd  KnoppMythR5C7.iso.33.15
577d93b66e07ac0f082c76cd67d99c8d  KnoppMythR5C7.iso.33.16
9d15bf221e4fa0dc2c7072958344e1a0  KnoppMythR5C7.iso.33.17
2c312db062abd19369a3d1897d7e069c  KnoppMythR5C7.iso.33.18
7497613eee33ef328fbd3bf4c699f3ec  KnoppMythR5C7.iso.33.19
101c6fe44d2a97b85f0fc5e6d91ff075  KnoppMythR5C7.iso.33.20
0876554bc3966ded2fbf3f8cb5391e4a  KnoppMythR5C7.iso.33.21
53ba3dd6f2938b1f824f65c2adc41308  KnoppMythR5C7.iso.33.22
d9afed65f8610347c047a8df07f98d68  KnoppMythR5C7.iso.33.23
65db148cc8aad5206a2dde4d3fe77423  KnoppMythR5C7.iso.33.24
954950312cddd54ed0d598240a5548c2  KnoppMythR5C7.iso.33.25
c6b8d8ce1c8946188355582a75c858fc  KnoppMythR5C7.iso.33.26
170045d3f1076b4f043dec7c3dbb4246  KnoppMythR5C7.iso.33.27
f10d9cd8bcd7d5a697ae453aa1cdb361  KnoppMythR5C7.iso.33.28
38795390b0d31fc7c5b3531858441cfd  KnoppMythR5C7.iso.33.29
f3a9650ff7ffb6f9c6e57e614d110404  KnoppMythR5C7.iso.33.30
060d1158f180ca7e8e381e8e80b11593  KnoppMythR5C7.iso.33.31
Author:  Ramrunner [ Thu Jun 08, 2006 9:14 pm ]
Post subject: 
We're down to about 5Gb.

The culprit is

e48966d03eb0dba101ecffa740f61fc0 KnoppMythR5C7.iso.33.10

This is the only one that's wrong at my end.

Would that be too big to send to my gmail address?

Regards,
Ed.
Author:  tjc [ Thu Jun 08, 2006 10:18 pm ]
Post subject: 
Still .5Mb, lets cut it one more time, down to 16Kb at which point I can uuencode it and PM you.
Code:
[tjc@silver KnoppMythR5C7.chunks]$ split -d -b 16384 KnoppMythR5C7.iso.33.10 KnoppMythR5C7.iso.33.10.
[tjc@silver KnoppMythR5C7.chunks]$ md5sum KnoppMythR5C7.iso.33.10.*
da70a5ffe36ae910e3cb635eba5af596  KnoppMythR5C7.iso.33.10.00
5c111122e198ef82945ac2ca0c1bd660  KnoppMythR5C7.iso.33.10.01
acc2c77f23f91c045af19da0c587e4fb  KnoppMythR5C7.iso.33.10.02
27e9bf0c9e25184b07308239193f93db  KnoppMythR5C7.iso.33.10.03
229a1034ce71f5bae4a15c89e56b8476  KnoppMythR5C7.iso.33.10.04
8e6539d72045ae38a7ff9eb685ba0e03  KnoppMythR5C7.iso.33.10.05
089c7a3a958d2e2557181fcc0c793aa3  KnoppMythR5C7.iso.33.10.06
5e743dfad0a706f202441552be701097  KnoppMythR5C7.iso.33.10.07
e2b37e9a1e9f9e55c8a72d32f153e1dc  KnoppMythR5C7.iso.33.10.08
db4ddb09417367fa08554080d8e70c9c  KnoppMythR5C7.iso.33.10.09
a26295ce6e82d69da0bc3b60387f9f3d  KnoppMythR5C7.iso.33.10.10
c50ba868fcafeb129cc55c5de56f8c8a  KnoppMythR5C7.iso.33.10.11
552a0e287c67a184a3e84d84f9c44555  KnoppMythR5C7.iso.33.10.12
afc724c609e6d02460f8ea64c867d5ea  KnoppMythR5C7.iso.33.10.13
276dd7036823eb55855b2bc6e71ac78c  KnoppMythR5C7.iso.33.10.14
5ccb1d877e8ac22c2baf9453297d6a43  KnoppMythR5C7.iso.33.10.15
0d56df606fe3092de975649751a042b3  KnoppMythR5C7.iso.33.10.16
6ee4849cc61aee5c922746072b17c4c0  KnoppMythR5C7.iso.33.10.17
3717096e842646c00df529a3ed6b8d26  KnoppMythR5C7.iso.33.10.18
c09051a1a08b11e0ee8cec5e7408844b  KnoppMythR5C7.iso.33.10.19
395d84bcb375c6fe104f4cb9bce47ff0  KnoppMythR5C7.iso.33.10.20
9be44c997a8b4e006e5de148ef7cbecf  KnoppMythR5C7.iso.33.10.21
4846dfbecc79a45c29144edb6f6edaf5  KnoppMythR5C7.iso.33.10.22
a9cc2442695c6e93d02cb6352cd05edc  KnoppMythR5C7.iso.33.10.23
c81e578610ad2386ff449f3e4b9e9a4e  KnoppMythR5C7.iso.33.10.24
58c22e5e1dac056bd0f6c8acbc14fa43  KnoppMythR5C7.iso.33.10.25
823490c4fd349f190e2883411ba0177d  KnoppMythR5C7.iso.33.10.26
0cac49c7a33a9c6381ba9880a0b451cc  KnoppMythR5C7.iso.33.10.27
0343e4f3941139bfc1c3d718306f7840  KnoppMythR5C7.iso.33.10.28
7d6b449bbc3e55e021a3005bbfa96682  KnoppMythR5C7.iso.33.10.29
5acdbca95f2f1082d0e182673b954d5b  KnoppMythR5C7.iso.33.10.30
6692b3111b4f44194ba2fea104907e73  KnoppMythR5C7.iso.33.10.31
Author:  Ramrunner [ Thu Jun 08, 2006 11:48 pm ]
Post subject: 
Got it:

229a1034ce71f5bae4a15c89e56b8476 KnoppMythR5C7.iso.33.10.04

Is now the only bad chunk.

Regards,
Ed.
Author:  Ramrunner [ Fri Jun 09, 2006 8:36 pm ]
Post subject: 
YES!!!

Thank you kind sir.

I managed to uudecode and get a prefect MD5SUM.

Then it was just a matter of catting the files together a section at a time, while checking the MD5SUMs in between.

I now have a perfect copy of R5C7 with a matching checksum.

Thank you for your time, you've saved me another 30 hour download.

Beer in the fridge if you're ever in Perth, Western Australia.

Regards,
Ramrunner
Page 1 of 1 All times are UTC - 6 hours
Powered by phpBB® Forum Software © phpBB Group
http://www.phpbb.com/