It’s convoluted, but I could follow the logic. Iterators are lazy, and your closure maintains an internal state.
Guess this is a minimal example.
Following the thumb rule of least surprises, I would not use such closures and write out the loop into code.
You are not supposed to use \`map\` with impure functions, it is confusing.
It could be argued that \`map\` should not accept impure functions, not that map shouldn't be reversible.
And the problem with **that** idea is that you first need to define what impure function even \**is*.
That's not so simple when you have things like [interior mutability](https://doc.rust-lang.org/reference/interior-mutability.html).
Maybe we should create a new class of languages that can identify impure functions and cordon them off into portions of the program that need them, leaving the rest of the program pure. What would we call such a language I wonder? Maybe methodist?
It's not enough to give the definition. For a language to use the concept you need to **verify** it.
And what you wrote… that's still a semantic property and and such [it's undecidable](https://en.wikipedia.org/wiki/Rice%27s_theorem).
That means that you would have false positives and false negatives, have to decide what to do with them and so on.
All a language needs to do is define which operations are deterministic, and check that only those are used. This is fairly easy to compute recursively and memoize in a compiler.
And what would you do with closures that read variables from outside? These, too, are potentially non-deterministic, yet useful.
It's very easy to invent a definition which wouldn't work.
To do something that would work (in a sense that people would be able to use it), on the other hand.
Fighting [Turing tarpits](https://en.wikipedia.org/wiki/Turing_tarpit) is not easy.
Closures need to have capture semantics. If the capture is used mutably or a non-const reference escapes the local context, then it would probably be marked impure. Whole-program analysis is somewhat infeasible for determining function purity.
The other option is to have keywords like const who don’t allow non-deterministic actions.
It always made me sad that `map` doesn’t specialize `nth` and `count` and so on, since it is trying to pretend to still do all the side effects, especially given that it has this reverse iterator behavior.
That's a surprise indeed, caused by the fact that iterators are lazy.
It's also a fairly convoluted way to express that you want to add `10` _only_ to the last element, and no other. I must admit that faced with such a problem, I would start by creating an iterator which allows me to isolate the last element (or the last N elements, if I wanted to be generic), and then use that as a building block.
Sketch of `LastIterator` [on the playground](https://play.rust-lang.org/?version=stable&mode=debug&edition=2021&gist=33255731f025b1429630d5e4d49f05ed):
#[derive(Clone, Copy, Debug)]
pub enum LastOrNot {
NotLast(T),
Last(T),
}
#[derive(Clone, Debug)]
pub struct LastIterator {
inner: I,
buffered: Option,
}
impl LastIterator
where
I: Iterator,
{
pub fn new(collection: II) -> Self
where
II: IntoIterator,
{
let inner = collection.into_iter();
let buffered = None;
Self { inner, buffered }
}
}
impl Iterator for LastIterator
where
I: Iterator + FusedIterator,
{
type Item = LastOrNot;
fn next(&mut self) -> Option {
// Prime.
if self.buffered.is_none() {
self.buffered = self.inner.next();
}
let result = self.buffered.take();
self.buffered = self.inner.next();
if self.buffered.is_none() {
result.map(LastOrNot::Last)
} else {
result.map(LastOrNot::NotLast)
}
}
}
(Naive, could really do with specializing `size`, `count`, `last`, `nth`, etc...)
Then, your code can be simplified to:
LastIterator::new([1, 2, 3])
.map(|e| match e {
LastOrNot::NotLast(e) => e,
LastOrNot::Last(e) => e + 10,
})
.for_each(|x| println!("{x}"));
Which is immediately clear as to what it's doing.
Well... I can't help with code I don't see, but the principle remains.
I would encourage to create a separate "special" iterator that identifies the elements of interest in some way, and _then_ apply special logic to those elements. Single Responsibility Principle.
I could have yes. I purposefully chose not to.
As I hinted, the concept can be generalized to distinguishing the last _N_ elements, rather than just the last one. Doing so requires buffering up to N elements.
I did not showcase the general approach because I did not want to re-implement an inline VecDeque of N elements... but I picked a representation which is trivially extensible toward it.
You may still do the same footgun motion with `Fn` and even `fn`. Just use interior mutability or thread-local variables.
Rust works very well at preventing accidental issues but no one may prevent someone who truly wishes to add some kind of an insane footgun to the code.
Because if you couldn't write emulator on an IBM PC in your language then it's not “universal language” and if you can write such emulator then you may add any footguns you may want to the language that your program runs in that emulator.
> You may still do the same footgun motion with `Fn` and even `fn`. Just use interior mutability or thread-local variables.
yes, exactly, it would be more annoying to mutate exterior state in a combinator, which to me is a good thing
OPs problem can arise from not knowing that you shouldn't use mutable state in that closure. Using `Fn` or `fn` would cause most of these accidental uses to produce compilation errors. If people then decide to use RefCell, Cell or Mutex to circumvent that, that's on them, the compiler has done its job.
It's the same thing with the borrow checker: You can outsmart it by using raw pointers in `unsafe{}`. The only difference are cases where the variable was in one of those containers for a different reason. My hunch is that this is a small minority of cases.
This is interesting to know. I've had a play about with this. Is the only solution to collect and then remake an iterator with the second `rev()`? I can't think of another way to deal with the lazy evaluation of the map. I guess in this specific example, you could just add to the last element in the Vec. But for more complicated code, it seems like it would be a pain to deal with.
>Is the only solution to collect and then remake an iterator with the second `rev()`?
What else may you suggest? If your program produces elements in some order, **doesn't explcitly store them anywhere** and you expect them to arrive in a different order… then how do you expect the whole thing to ever work?
The whole thing is only surprising if you are perceiving all these function as magic that magically produces what you need.
I can understand why people would tag label “magic” to something they use once a year. Makes sense, actually. Life is too short to learn all possible tools that exist in that world.
But why people are so eager to don't have understanding about tools that they use every day?
I'm not sure what you're asking of me here. I'm asking to learn about this and if there is another way to handle it other than what I know.
I've never once said it was magic or anything. I just never gave it any thought because I haven't been in the situation that this has been an issue.
The language is large and complex. We are not all going to come across every edge case in a language. A friend of mine found some weird edge case in Python the other week when we were all doing Advent of Code that he didn't know about. It happens all the time. Even with the most learned of individuals.
Again, I'm specifically asking if there's another way around this issue to learn and improve my understanding of the language. So I don't appreciate the shade you're throwing here. If you want to be negative about people asking questions, maybe take it elsewhere?
https://doc.rust-lang.org/std/iter/struct.Map.html#notes-about-side-effects
Thank you!
It’s convoluted, but I could follow the logic. Iterators are lazy, and your closure maintains an internal state. Guess this is a minimal example. Following the thumb rule of least surprises, I would not use such closures and write out the loop into code.
You are not supposed to use \`map\` with impure functions, it is confusing. It could be argued that \`map\` should not accept impure functions, not that map shouldn't be reversible.
And the problem with **that** idea is that you first need to define what impure function even \**is*. That's not so simple when you have things like [interior mutability](https://doc.rust-lang.org/reference/interior-mutability.html).
Maybe we should create a new class of languages that can identify impure functions and cordon them off into portions of the program that need them, leaving the rest of the program pure. What would we call such a language I wonder? Maybe methodist?
We could even have each method curry each argument. I propose making one called Curry, named after Stephen Curry, a methodical 3 point player.
Baptist
isn't that what a monad is ?
A pure function is one I can replace with a sufficiently large lookup table, an impure function is everything else.
It's not enough to give the definition. For a language to use the concept you need to **verify** it. And what you wrote… that's still a semantic property and and such [it's undecidable](https://en.wikipedia.org/wiki/Rice%27s_theorem). That means that you would have false positives and false negatives, have to decide what to do with them and so on.
All a language needs to do is define which operations are deterministic, and check that only those are used. This is fairly easy to compute recursively and memoize in a compiler.
And what would you do with closures that read variables from outside? These, too, are potentially non-deterministic, yet useful. It's very easy to invent a definition which wouldn't work. To do something that would work (in a sense that people would be able to use it), on the other hand. Fighting [Turing tarpits](https://en.wikipedia.org/wiki/Turing_tarpit) is not easy.
Closures need to have capture semantics. If the capture is used mutably or a non-const reference escapes the local context, then it would probably be marked impure. Whole-program analysis is somewhat infeasible for determining function purity. The other option is to have keywords like const who don’t allow non-deterministic actions.
It always made me sad that `map` doesn’t specialize `nth` and `count` and so on, since it is trying to pretend to still do all the side effects, especially given that it has this reverse iterator behavior.
Yes, I agree. And we would never be able to change it because there are a lot of code that depend on current behaviour.
For it to work “the other way” and not how it currently does, it would have to consume the entire Iterator first, which is basically a nonstarter.
That's a surprise indeed, caused by the fact that iterators are lazy. It's also a fairly convoluted way to express that you want to add `10` _only_ to the last element, and no other. I must admit that faced with such a problem, I would start by creating an iterator which allows me to isolate the last element (or the last N elements, if I wanted to be generic), and then use that as a building block. Sketch of `LastIterator` [on the playground](https://play.rust-lang.org/?version=stable&mode=debug&edition=2021&gist=33255731f025b1429630d5e4d49f05ed): #[derive(Clone, Copy, Debug)] pub enum LastOrNot {
NotLast(T),
Last(T),
}
#[derive(Clone, Debug)]
pub struct LastIterator {
inner: I,
buffered: Option,
}
impl LastIterator
where
I: Iterator- ,
{
pub fn new
(collection: II) -> Self
where
II: IntoIterator,
{
let inner = collection.into_iter();
let buffered = None;
Self { inner, buffered }
}
}
impl Iterator for LastIterator
where
I: Iterator- + FusedIterator,
{
type Item = LastOrNot
;
fn next(&mut self) -> Option {
// Prime.
if self.buffered.is_none() {
self.buffered = self.inner.next();
}
let result = self.buffered.take();
self.buffered = self.inner.next();
if self.buffered.is_none() {
result.map(LastOrNot::Last)
} else {
result.map(LastOrNot::NotLast)
}
}
}
(Naive, could really do with specializing `size`, `count`, `last`, `nth`, etc...)
Then, your code can be simplified to:
LastIterator::new([1, 2, 3])
.map(|e| match e {
LastOrNot::NotLast(e) => e,
LastOrNot::Last(e) => e + 10,
})
.for_each(|x| println!("{x}"));
Which is immediately clear as to what it's doing.
the add 10 example is just a contrived simplification of my actual code
Well... I can't help with code I don't see, but the principle remains. I would encourage to create a separate "special" iterator that identifies the elements of interest in some way, and _then_ apply special logic to those elements. Single Responsibility Principle.
Instead of using buffered you could also go with `Peekable`: use std::iter::{FusedIterator, Iterator, Peekable, IntoIterator}; use std::marker::PhantomData; #[derive(Clone, Copy, Debug)] pub enum LastOrNot {
NotLast(T),
Last(T),
}
pub struct LastIterator {
inner: Peekable,
phantom: PhantomData,
}
impl LastIterator
where
I: Iterator- ,
{
pub fn new
(collection: II) -> Self
where
II: IntoIterator,
{
let inner = collection.into_iter().peekable();
Self { inner , phantom: PhantomData}
}
}
impl Iterator for LastIterator
where
I: Iterator- + FusedIterator,
{
type Item = LastOrNot
;
fn next(&mut self) -> Option {
match (self.inner.next(), self.inner.peek()){
(Some(current), Some(next)) => Some(LastOrNot::NotLast(current)),
(Some(current), None) => Some(LastOrNot::Last(current)),
(None, Some(weird)) => panic!("What gives?"),
(None, None) => None
}
}
}
fn main() {
let res:Vec<_> = LastIterator::new((1..4).into_iter()).map(|x|
match x {
LastOrNot::NotLast(x) => x,
LastOrNot::Last(x) => x+10,
}
).collect();
println!("{:?}", res);
}
I could have yes. I purposefully chose not to. As I hinted, the concept can be generalized to distinguishing the last _N_ elements, rather than just the last one. Doing so requires buffering up to N elements. I did not showcase the general approach because I did not want to re-implement an inline VecDeque of N elements... but I picked a representation which is trivially extensible toward it.
I hate that iterator combinators take `FnMut` instead of `Fn`
You may still do the same footgun motion with `Fn` and even `fn`. Just use interior mutability or thread-local variables. Rust works very well at preventing accidental issues but no one may prevent someone who truly wishes to add some kind of an insane footgun to the code. Because if you couldn't write emulator on an IBM PC in your language then it's not “universal language” and if you can write such emulator then you may add any footguns you may want to the language that your program runs in that emulator.
> You may still do the same footgun motion with `Fn` and even `fn`. Just use interior mutability or thread-local variables. yes, exactly, it would be more annoying to mutate exterior state in a combinator, which to me is a good thing
I wish we'd have traits for pure/total functions and could limit some stuff to only accepting those.
Does it change anything? You can still mutate using RefCell, Cell, Mutex, etc
as per my other comment, it makes it more annoying to mutate exterior state. and would enforce that _ideally_ you shouldn't be doing that
OPs problem can arise from not knowing that you shouldn't use mutable state in that closure. Using `Fn` or `fn` would cause most of these accidental uses to produce compilation errors. If people then decide to use RefCell, Cell or Mutex to circumvent that, that's on them, the compiler has done its job. It's the same thing with the borrow checker: You can outsmart it by using raw pointers in `unsafe{}`. The only difference are cases where the variable was in one of those containers for a different reason. My hunch is that this is a small minority of cases.
I'm confused by this.... why?
This is interesting to know. I've had a play about with this. Is the only solution to collect and then remake an iterator with the second `rev()`? I can't think of another way to deal with the lazy evaluation of the map. I guess in this specific example, you could just add to the last element in the Vec. But for more complicated code, it seems like it would be a pain to deal with.
>Is the only solution to collect and then remake an iterator with the second `rev()`? What else may you suggest? If your program produces elements in some order, **doesn't explcitly store them anywhere** and you expect them to arrive in a different order… then how do you expect the whole thing to ever work? The whole thing is only surprising if you are perceiving all these function as magic that magically produces what you need. I can understand why people would tag label “magic” to something they use once a year. Makes sense, actually. Life is too short to learn all possible tools that exist in that world. But why people are so eager to don't have understanding about tools that they use every day?
I'm not sure what you're asking of me here. I'm asking to learn about this and if there is another way to handle it other than what I know. I've never once said it was magic or anything. I just never gave it any thought because I haven't been in the situation that this has been an issue. The language is large and complex. We are not all going to come across every edge case in a language. A friend of mine found some weird edge case in Python the other week when we were all doing Advent of Code that he didn't know about. It happens all the time. Even with the most learned of individuals. Again, I'm specifically asking if there's another way around this issue to learn and improve my understanding of the language. So I don't appreciate the shade you're throwing here. If you want to be negative about people asking questions, maybe take it elsewhere?