foreach

[pkoppstein]

I was reading the release notes for 0.9.0 and was quite disappointed to read about some of the changes affectingforeach in the BREAKING CHANGE notes.

Obviously breaking changes can often be justified, and discrepancies from jq are to be expected, but some of the changes affecting foreach are, it seems to me, very hard to justify given (a) their utility,(b) the ordinary meaning of "for each" in English.; and (c) the fact that gojq also conforms with the jq semantics.

Since 0.9.0 < 1.0, I am hoping that you will reconsider at least some aspects of foreach, e.g. its name.

For example, since you evidently feel strongly about the init value being emitted, one possibility would be for jaq to have the control structure you want under a different name. Let's suppose it was named for. Then (ideally perhaps) we could have our cake and eat it (i.e. have for and foreach), but if you do not wish to have both, then having your control structure as for would at least help avoid confusion.

By the way, congratulations on all the improvements in 0.9.0!

[01mf02]

Obviously breaking changes can often be justified, and discrepancies from jq are to be expected, but some of the changes affecting foreach are, it seems to me, very hard to justify given (a) their utility,(b) the ordinary meaning of "for each" in English.; and (c) the fact that gojq also conforms with the jq semantics.

After reflection, I agree with you.

For example, since you evidently feel strongly about the init value being emitted, one possibility would be for jaq to have the control structure you want under a different name. Let's suppose it was named for. Then (ideally perhaps) we could have our cake and eat it (i.e. have for and foreach), but if you do not wish to have both, then having your control structure as for would at least help avoid confusion.

I like your idea of having for. The shorter name suggests that it is preferred (for theoretical and performance reasons), but you can keep using foreach if you need compatibility with jq.
I implemented this approach in 26b0c39. Feel free to tell me if you find issues with it.

By the way, congratulations on all the improvements in 0.9.0!

Thank you! I already have some plans for the next release (no breaking changes planned this time ^^).

[pkoppstein]

@01mf02 - Fantastic! Thanks for the update, and thanks for being willing to reconsider.

[pkoppstein]

@01mf02 - Your most recent comment in this thread suggested you intended to support for as well as
a (go)jq-compatible foreach, but in version 772879b,
foreach/3 (i.e. the version with three parts in the parentheses) is not supported (see below).

To be clear, in jq:

jq -n 'foreach (1,2) as $x (0; .+$x; if $x == 1 then empty else . end)'
3

The reason foreach/3 is very helpful is of course that the iteration
variable ($x in this example) can be used in the third part.
Otherwise, we'd have to write something awkward like:

jq -n 'foreach (1,2) as $x ({y:0}; {x:$x, y: (.y + $x)}) | if .x == 1 then empty else .y end'

###################################

$ jaq -n 'foreach (1,2) as $x (0; .+$x)'
1
3
$ jaq -n 'foreach (1,2) as $x (0; .+$x)'
1
3
$ jaq -n 'for (1,2) as $x (0; .+$x)'
0
1
3

$ jaq -n 'foreach (1,2) as $x (0; .+$x; if $x == 1 then empty else . end)'
Error: Unexpected token, expected ?, -=, *=, *, ), |=, or, <=, -, +, [, //, ,, %=, as, |, and, >, +=, /=, !=, ==, =, >=, /, <, %, .
   ╭─[<unknown>:1:29]
   │
 1 │ foreach (1,2) as $x (0; .+$x; if $x == 1 then empty else . end)
   ·                             ┬  
   ·                             ╰── Unexpected token ;
───╯
$ 

[01mf02]

Sorry for the misunderstanding --- indeed, I only wanted to suggest that I wanted to support some subset of foreach that is compatible with jq (namely foreach/2), but this subset does not include foreach/3.
Why is that so? In a nutshell, foreach/3 is a very annoying special case to handle. It requires completely separate logic from foreach/2 and reduce in both the parser and in the interpreter. In addition, as you mentioned, it can be emulated with foreach/2. Is it awkward? Perhaps a bit, but only if you really need access to $x. Without that, this can be done equally compactly by filtering outside of foreach. And how often do you actually need access to $x in foreach/3? Does it justify the greater complexity of the implementation?
(On the side, it would be great to have a large repository of jq scripts that people actually use in the wild. This would allow me to focus on features that most people actually use. And it would allow us, for example, to study how many people use $x in foreach/3. So far, I just look at scripts that I happen to find by accident, and in these, usage of foreach/3 is pretty rare.)

Also, I think it is that foreach/2 is not that awkward after all. Let me show you some code that does the same thing, but in different ways.

1. The first one is what you wrote to illustrate that foreach/2 is awkward.
2. The second one is a shortened version I wrote using foreach/2.
3. The third one is what you wrote to illustrate that foreach/3 is more compact.
4. The fourth one is what I wrote as a most compact way using foreach/3.
5. The fifth one uses the new for instead of foreach/2.

1. 'foreach (1,2) as $x ({y:0}; {x:$x, y: (.y + $x)}) | if .x == 1 then empty else .y end'
2. 'foreach (1,2) as $x ({}; {x: $x, y: .y + $x}) | select(.x == 1) | .y'
3. 'foreach (1,2) as $x (0; . + $x; if $x == 1 then empty else . end)'
4. 'foreach (1,2) as $x (0; . + $x; select($x == 1))'
5. 'for (1,2) as $x ({}; {x: $x, y: .y + $x}) | select(.x == 1) | .y'

So here, foreach/3 (4) is indeed the shortest version. However, using foreach/2 (2), we are not that far away, and using for (5), this is even shorter than what you originally wrote (3).

In conclusion, I believe that the slightly shorter foreach/3 does not really justify the implementation complexity that comes with it.

[pkoppstein]

it would be great to have a large repository of jq scripts that people actually use in the wild.

There is a large corpus at rosettacode.org: https://rosettacode.org/wiki/Category:Jq

Currently there are 926 worked examples.

Unfortunately I can’t easily quantify how often the “special case” is used, but the genius of jq’s foreach/3 is indeed that it is not trivially reducible, and indeed the reason I’ve brought up the issue is that I have found it useful (and “essential” to avoid the circumlocution). Because of this and the issue of compatibility with jq/gojq (and earlier versions of jaq), I hope you will further reconsider, but in any case, thanks again for taking the time to reflect and explain.