Merged
Conversation
bitonic
force-pushed
the
linear-getByteString
branch
from
bea0981 to
b6cf044
Compare
Contributor
Author
|
Note that this is probably better done with a |
bitonic
force-pushed
the
linear-getByteString
branch
from
b6cf044 to
bf8eb3e
Compare
Chains of `B.append`s were being created by repeated calls to `demandInput`. Try the following program, which writes and read 100MB, to appreciate the difference: ``` import qualified Data.ByteString as BS import qualified Data.ByteString.Lazy as BSL import Data.Binary (encode, decode) import Data.Char (ord) main :: IO () main = do let inBs = BS.replicate 100000000 (fromIntegral $ ord 'a') BSL.writeFile "bs.bin" (encode inBs) putStrLn "writing done" bin <- BSL.readFile "bs.bin" -- This takes around 30 seconds and causes more than 10GB to be -- allocated. let outBs = decode bin print $ inBs == outBs ```
bitonic
force-pushed
the
linear-getByteString
branch
from
bf8eb3e to
e46b7b6
Compare
Member
|
Hey! Thanks for the fix. Indeed, there's a problem with I'm not sure why a |
Contributor
Author
|
Wasn't the previous |
Member
|
Part of it was inlineable. That part is used whenever there is enough input (99% of the time). |
...by making the common case (when the input is big enough) non-recursive.
Contributor
Author
|
I pushed a commit that makes |
Member
|
Thanks! I'll have a look later today. |
hvr
added a commit
to ghc/ghc
that referenced
this pull request
Jun 1, 2015
Quoting the changelog, this pulls in the following fixes:
binary-0.7.5.0
--------------
- Fix performance bug that was noticable when you get a big strict ByteString
and the input to the decoder consists of many small chunks.
- haskell/binary#73
- haskell/binary#76
- Fix memory leak when decoding Double and Float.
- Commit 497a181c083fa9faf7fa3aa64d1d8deb9ac76ecb
- We now require QuickCheck >= 2.8. Remove our version of arbitrarySizedNatural.
binary-0.7.4.0
--------------
- Some invalid UTF-8 strings caused an exception when decoded. Those errors will
now now fail in the Get monad instead. See issue 70.
Patch contributed by @ttuegel.
hvr
added a commit
to ghc/ghc
that referenced
this pull request
Jun 9, 2015
Quoting the changelog, this pulls in the following fixes:
binary-0.7.5.0
--------------
- Fix performance bug that was noticable when you get a big strict ByteString
and the input to the decoder consists of many small chunks.
- haskell/binary#73
- haskell/binary#76
- Fix memory leak when decoding Double and Float.
- Commit 497a181c083fa9faf7fa3aa64d1d8deb9ac76ecb
- We now require QuickCheck >= 2.8. Remove our version of arbitrarySizedNatural.
binary-0.7.4.0
--------------
- Some invalid UTF-8 strings caused an exception when decoded. Those errors will
now now fail in the Get monad instead. See issue 70.
Patch contributed by @ttuegel.
(cherry picked from commit 7dd0ea7)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
getByteStringexhibited quadratic behaviour -- a classic case of bad "appending". This commit fixes it.