Question 110616 in Json Question posted in
Our archive of expertly curated questions and answers provides insights and solutions to common problems related to this popular data interchange format. From parsing and manipulating JSON data to integrating it with various programming languages and web services, our archive has got you covered. Start exploring today and take your JSON skills to the next level

AppleScript doesn't complete when parsing large json structure

I have an iMac 2017 with Monterey 12.6.

I am forced to call my API in python to have a correct JSON structure for results.

For information, this is the URL that I call in python (coingecko limits the result to the 100 first results):

https://api.coingecko.com/api/v3/coins/markets?ids=acala,alpaca-finance,altair,astar,avalanche-2,baanx,bbeefy-finance,bifrost-native-coin,binancecoin,binance-eth,binance-usd,bitcoin,cardano,chainlink,chainx,chronicle,coin-capsule,colony,cosmos,crabada,crypto-com-chain,cumrocket,curve-dao-token,dappradar,dogecoin,elrond-erd-2,ergo,ethereum,evmos,exeedme,fantom,ftx-token,fuse-network-token,genshiro,green-satoshi-token,havven,hooked-protocol,integritee,kadena,karura,kintsugi,kucoin-shares,kusama,kuswap,matic-network,metagame-arena,metagods,metavault,mina-protocol,moonbeam,moonpot,moonriver,nafty,near,osmosis,pancakeswap-token,paraswap,platypus-finance,pluton,polkadot,safemoon,shiba-inu,kryll,kucoin-shares,kusama,kuswap,lido-dao,matic-network,maze-token,memepad,metagame-arena,metagods,metis-token,mina-protocol,moonbeam,moonpot,moonriver,movn,nafter,nafty,near,nexo,nftlaunch,orion-protocol,osmosis,paid-network,pancakeswap-token,paraswap,platypus-finance,polkadot,polkamon,polkamarkets,polkastarter,polycat-finance,polychain-monsters,polygonfarm-finance,presearch,safemoon,safepal,shiba-inu,shiden,solana,staked-ether,staked-olympus,stasis-eurs,stepn,sushi,swissborg,switch,tether,terrausd,the-graph,the-sandbox,unifarm,uniswap,usd-coin,valkyrie-protocol,wizarre-scroll,zelcash&vs_currency=EUR"

The result in Python is something like that but with many others cryptos:

[{'id': 'bitcoin', 'symbol': 'btc', 'name': 'Bitcoin', 'image': 'https://assets.coingecko.com/coins/images/1/large/bitcoin.png?1547033579', 'current_price': 15765.21, 'market_cap': 303279048538, 'market_cap_rank': 1, 'fully_diluted_valuation': 330971307514, 'total_volume': 9519061248, 'high_24h': 15812.98, 'low_24h': 15758.06, 'price_change_24h': -18.11016048005331, 'price_change_percentage_24h': -0.11474, 'market_cap_change_24h': -207406972.0255127, 'market_cap_change_percentage_24h': -0.06834, 'circulating_supply': 19242937.0, 'total_supply': 21000000.0, 'max_supply': 21000000.0, 'ath': 59717, 'ath_change_percentage': -73.60052, 'ath_date': '2021-11-10T14:24:11.849Z', 'atl': 51.3, 'atl_change_percentage': 30631.82104, 'atl_date': '2013-07-05T00:00:00.000Z', 'roi': None, 'last_updated': '2022-12-25T14:21:15.974Z'}, {'id': 'ethereum', 'symbol': 'eth', 'name': 'Ethereum', 'image': 'https://assets.coingecko.com/coins/images/279/large/ethereum.png?1595348880', 'current_price': 1142.38, 'market_cap': 137652680087, 'market_cap_rank': 2, 'fully_diluted_valuation': 137652680087, 'total_volume': 2165257956, 'high_24h': 1149.42, 'low_24h': 1141.8, 'price_change_24h': -1.0190823115187868, 'price_change_percentage_24h': -0.08913, 'market_cap_change_24h': -73228021.97866821, 'market_cap_change_percentage_24h': -0.05317, 'circulating_supply': 120523982.078808, 'total_supply': 120523982.078808, 'max_supply': None, 'ath': 4228.93, 'ath_change_percentage': -72.99273, 'ath_date': '2021-12-01T08:38:24.623Z', 'atl': 0.381455, 'atl_change_percentage': 299310.75605, 'atl_date': '2015-10-20T00:00:00.000Z', 'roi': {'times': 95.89474372830291, 'currency': 'btc', 'percentage': 9589.474372830291}, 'last_updated': '2022-12-25T14:20:55.699Z'}]

Content of my AppleScript:

set desktop_folder to "$HOME/PycharmProjects/crypto/"
set valReturned to do shell script "python3 " & desktop_folder & "crypto.py"

set coins to (every item in valReturned) as list
repeat with n from 1 to count of coins
    set coin to item n of coins
end repeat

The AppleScript process uses 99% of my processor.

If I comment on the loop part, I have a huge JSON (it’s just the result for 100 different cryptocurrencies). I can’t post it because when I copy-paste its contents, it exceeds 81,000 characters here.

Why does this crash AppleScript?

I’ve already posted on Stackoverflow here but I didn’t do it the same way.
I think I could decrease the number of cryptos in my request but it’s still strange that it crashes this way. I have no problem with Excel and powerquery for example.

I feel like the best way to produce a Numbers file would be to generate a CSV and then use AppleScript to import that CSV into Numbers and then apply formatting. Any suggestions?

Tags:

2

Answers


  1. 0

    Can only speculate, as I don’t have/use any Apple-anything:

    • There’s no linebreaks in the JSON. If loading/processing it with a tool/language that relies (internally) on strings, it may exceed a maximum character limit for a string
    • In both cases, your valReturned is a variable in AppleScript. It may come with some size limitations, and also you’re filling/setting it via the textual characters that come from the output of echo or pyhton3. This can lead to delays and size limitations (crashes?). Especially with echo, printing text on the terminal is often limited by the terminal’s output/drawing/rendering speed, not how fast the computer can process the characters.
    • Not sure how AppleScript handles JSON – does it parse the JSON and keep it all in memory? Also, does it try to be clever and unserialize/demarshal the JSON into its own data types, creating many objects etc. for later programmatic access? You seem to then also have two copies of it, one in a list structure in coins and then the valReturned (as a string I assume).
    • Potentially more similar reasons like that…

    If you’re already able to call echo and python3, maybe you do not need to use AppleScript to iterate over the JSON entries? As you seem to be able to write the JSON from the network into a file directly, maybe you do not need to have the AppleScript load/keep the JSON data? Wonder if trying to avoid AppleScript’s exposure to the JSON or results/output from the external programs would mitigate the performance + crash problems.

    Login or Signup to reply.
  2. 0

    AppleScript will perform the shell script without any issues, but note that the results will be a string. Your sample script is just going through the (81,000+) characters of that string, which takes a little time. Many languages (such as Python) provide direct support for JSON strings – for AppleScript, Cocoa (via AppleScriptObjC) can be used.

    The JSON string from the shell script can be parsed to get a list of records (the result can also be left as Cocoa objects for use with those methods). The individual list items can then stepped through using a repeat statement and the various record key values used as desired, for example:

    use framework "Foundation"
use scripting additions

set theURL to "https://api.coingecko.com/api/v3/coins/markets?ids=acala,alpaca-finance,altair,astar,avalanche-2,baanx,bbeefy-finance,bifrost-native-coin,binancecoin,binance-eth,binance-usd,bitcoin,cardano,chainlink,chainx,chronicle,coin-capsule,colony,cosmos,crabada,crypto-com-chain,cumrocket,curve-dao-token,dappradar,dogecoin,elrond-erd-2,ergo,ethereum,evmos,exeedme,fantom,ftx-token,fuse-network-token,genshiro,green-satoshi-token,havven,hooked-protocol,integritee,kadena,karura,kintsugi,kucoin-shares,kusama,kuswap,matic-network,metagame-arena,metagods,metavault,mina-protocol,moonbeam,moonpot,moonriver,nafty,near,osmosis,pancakeswap-token,paraswap,platypus-finance,pluton,polkadot,safemoon,shiba-inu,kryll,kucoin-shares,kusama,kuswap,lido-dao,matic-network,maze-token,memepad,metagame-arena,metagods,metis-token,mina-protocol,moonbeam,moonpot,moonriver,movn,nafter,nafty,near,nexo,nftlaunch,orion-protocol,osmosis,paid-network,pancakeswap-token,paraswap,platypus-finance,polkadot,polkamon,polkamarkets,polkastarter,polycat-finance,polychain-monsters,polygonfarm-finance,presearch,safemoon,safepal,shiba-inu,shiden,solana,staked-ether,staked-olympus,stasis-eurs,stepn,sushi,swissborg,switch,tether,terrausd,the-graph,the-sandbox,unifarm,uniswap,usd-coin,valkyrie-protocol,wizarre-scroll,zelcash&vs_currency=EUR"

set valReturned to (do shell script "/usr/bin/curl " & quoted form of theURL) -- string
set theResult to parseJSON from valReturned -- list of records

set output to "Current Prices:" & return -- header for the example
repeat with anItem in theResult -- the individual list items
   -- do whatever with the record key values
   set theName to |name| of anItem -- "name" is a keyword
   set theValue to current_price of anItem
   set output to output & theName & ": " & theValue & return
end repeat
return output

# Parse a JSON string into a data structure.
to parseJSON from sourceString given coercion:coerce : true
   if class of sourceString is not in {string} then error "parseJSON error: source is not a string"
   set theString to current application's NSString's stringWithString:sourceString
   set theData to theString's dataUsingEncoding:(current application's NSUTF8StringEncoding)
   set {theObject, theError} to current application's NSJSONSerialization's JSONObjectWithData:theData options:0 |error|:(reference)
   if theObject is missing value then error "parseJSON error: " & (theError's userInfo's objectForKey:"NSDebugDescription")
   if coerce then if (theObject's isKindOfClass:(current application's NSArray)) as boolean then
      return theObject as list
   else
      return theObject as record
   end if
   return theObject -- leave as NSArray or NSDictionary
end parseJSON
    Login or Signup to reply.
Please signup or login to give your own answer.

Back To Top
Search