How can information (Shannon) entropy decrease ?

[swansont]

3 hours ago, SuperSlim said:

What a pack of retards.

!

Moderator Note

Personal attacks and slurs such as this will not be tolerated. 

Stick to discussing the subject

 

[Ghideon]

5 hours ago, SuperSlim said:

According to you a computer can be switched off and still be computing! 

Your conclusion is, intentionally or unintentionally, incorrect.  

5 hours ago, SuperSlim said:

My guess is you probably think data and information are different things too.

Your guess is incorrect. Data and information are different concepts and the differences are addressed differently depending on the context of discussion. Sorting out the details may be better suited for a separate thread.

5 hours ago, SuperSlim said:

You provide an example: the Swedish language without the extra marks. A change of encoding that makes almost no difference to the information content.

A quick example, here are three different Swedish phrases with very different meaning*. Only the dots differ: 

får får får
får far får
far får får

A fourth sentence with a completely different meaning:

far far far

Without the dots the first three examples and the last example are indistinguishable and that has impact on the entropy.

5 hours ago, SuperSlim said:

So it has about the same entropy.

The example addresses the initial general question about decreasing entropy. "About the same" is too vague to be interesting in this context.

 

5 hours ago, SuperSlim said:

completely dumbass

I take that and similar entries as an illustration of entropy as defined by Shannon; the redundancy in the texts allows for this to be filtered out without affecting the discussion. And if the level of noise is too high from a specific sender it may be blocked or disconnected from the channel altogether.

 

*) (approximate) translations of the four examples
1: does sheep give birth to sheep
2: does father get sheep
3: father gets sheep
4: go father go

Edited March 15, 2022 by Ghideon
Translations added

[SuperSlim]

3 hours ago, Ghideon said:

I take that and similar entries as an illustration of entropy as defined by Shannon; the redundancy in the texts allows for this to be filtered out without affecting the discussion. And if the level of noise is too high from a specific sender it may be blocked or disconnected from the channel altogether.

 

That looks almost intelligent. What you've clearly forgotten is that languages are context-dependent.

Strange too, that after dismissing the need for a sender, receiver and a channel, you invoke the concept of noise in a channel, and filtering.

You don't realise how inane that is. You don't because you have immunity, right?

The difference between information and data: there is no physical difference, it's entirely artificial; it's one of those things called a choice. You can't or won't agree of course, because this discussion is all about how much you can disagree with whatever you choose to disagree with. What fun you must be having,

Edited March 15, 2022 by SuperSlim

[Ghideon]

On 3/13/2022 at 4:23 PM, joigus said:

I would seriously would like this conversation to get back on its tracks.

I agree. 

On 3/13/2022 at 4:23 PM, joigus said:

I think entropy can be defined at many different levels depending on the level of description that one is trying to achieve. In that sense, I think it would be useful to talk about control parameters, which I think say it all about what level of description one is trying to achieve.

Every system (whether a computer, a gas, or a coding machine) would have a set of states that we can control, and a set of microstates, that have been programmed either by us or by Nature, that we can't see, control, etc. It's in that sense that the concept of entropy, be it Shannon's or Clausius/Boltzmann, etc. is relevant.

I (yet) lack the knowledge about the role of control parameters so I'll make a general reflection for possible further discussion. When calculating Shannon entropy in the context of computer science the level of interest is usually logical. For instance your calculations of the coin example did not need to bother about the physical coins and papers to get to a correct result per the definition of entropy. Time is not something that affect the solution*. Physical things on the other hand such as papers, computers, and storage devices do of course decay; even if it may take considerable time the life span is finite. Does this make sense in the context of levels of description above?

If so, then we may say that the computer program cannot read out exact information about each physical parameter that will cause such a failure. Failure in this case means that without external intervention (spare parts or similar) the program halts, returns incorrect data or similar bond what builtin fault tolerance is capable of handling. Does this relate to your following statement?

On 3/13/2022 at 4:23 PM, joigus said:

It's my intuition that in the case of a computer, the control parameters are the bits that can be read out, while the entropic degrees of freedom correspond to the bits that are being used by the program, but cannot be read out --thereby the entropy. But I'm not sure about this and I would like to know of other views on how to interpret this.

 

Note: I've probed at a physical meaning of "cannot read out" in my above answer; an area I'm less familiar with but your comments triggers my curiosity. There are other possible aspects; feel free to steer the discussion towards what interests you @joigus.

 

 

On 3/13/2022 at 4:23 PM, joigus said:

The fact that Shannon entropy may decrease doesn't really bother me because, as I said before, a system that's not the whole universe can have its entropy decrease without any physical laws being violated.

I will think about this for a while before answering; there might be interesting aspects to this from a practical point of view.  

 

 

5 hours ago, SuperSlim said:

What fun you must be having

That seems like a valid conclusion; I would end my participation in this thread if it was not fun.

 

*) Neglecting the progress of questions and answers; each step, once calculated, do not change. 

Edited March 15, 2022 by Ghideon
clarification

[joigus]

1 hour ago, Ghideon said:

Time is not something that affect the solution*.

 

1 hour ago, Ghideon said:

*) Neglecting the progress of questions and answers; each step, once calculated, do not change. 

I assume you mean that @studiot's system doesn't really change its entropy? Its state doesn't really change, so there isn't any dynamics in that system? It's the computing system that changes its entropy by incrementally changing its "reading states." After all, the coin is where it is, so its state doesn't change; thereby its entropy doesn't either. Is that what you mean?

Please, give me some more time to react to the rest of your comments, because I think a bridge can be built between the physical concept of entropy, of which I know rather well, to the computer people like you, of which I'm just trying to understand better.

Thank you for your inexhaustible patience, @Ghideon.

I try to keep my entropy constant, but it's not easy.

[Ghideon]

21 hours ago, joigus said:

I assume you mean that @studiot's system doesn't really change its entropy? Its state doesn't really change, so there isn't any dynamics in that system? It's the computing system that changes its entropy by incrementally changing its "reading states." After all, the coin is where it is, so its state doesn't change; thereby its entropy doesn't either. Is that what you mean?

Thanks for your input. I'll try to clarify by using four examples based on my current understanding. Information entropy in this case means the definition from Shannon. By physical entropy I mean any suitable definition from physics*; here you may need to fill in the blanks or highlight where I may have misunderstood** things.

1: Assume information entropy is calculated as per Shannon for some example. In computer science we (usually) assume an ideal case; physical implementation is abstracted away. Time is not part of the Shannon definition and physics plays no part in the outcome of entropy calculation in this case.

2: Assume we store the input parameters and/or the result of the calculation from (1) in digital form. In the ideal case we also (implicitly) assume unlimited lifetime of the components in computers or unlimited supply of spare parts, redundancy, fault tolerance and error correction so that the mathematical result from (1) still holds; the underlaying physics have been abstracted away by assuming nothing ever breaks or that any error can be recovered from. In this example there is some physics but under the assumptions made the physics cannot have an effect on the outcome.

3: Assume we store the result of the calculation from (1) in digital form on a real system (rather than modelling an ideal system). The lifetime of the system is not unlimited and at some future point the results from (1) will be unavailable or if we try to repeat the calculation based on the stored data we may get a different result. We have moved from the ideal computer science world (where I usually dwell) into an example where the ideal situation of (1) and (2) does not hold. In this 3rd case my guess is that physics, and physical entropy, play a part. We loose (or possibly get incorrect) digital information due to faulty components or storage and this have impact on the Shannon entropy for the bits we manage to read out or calculate. The connection to physical entropy here is one of the things I lack knowledge about but I'm curious about. 

4: Assume we store the result of the calculation from (1) in digital form on an ideal system (limitless lifetime) using lossy compression****.  This means that at a later state we cannot repeat the exact calculation or expect identical outcome since part of the information is lost and cannot be recovered by the digital system. In this case we are still in the ideal world of computer science where the predictions or outcome is determined by computer science theorems. Even if there is loss of information physics is still abstracted away and physical entropy plays no part.

Note here the similarities between (3) and (4). A computer scientist can analyse the information entropy change and the loss of information due to a (bad) choice of compression in (4). The loss of information in (3) due to degrading physical components seems to me to be connected to physical entropy. Does this make sense? If so:

On 3/4/2022 at 11:08 PM, joigus said:

It's very tempting to me to start talking about control parameters and how they really determine what information is available to anyone trying to describe a system, but it would make it a discussion to heavily imbued with a purely-physics outlook.

It would be interesting to see where control parameters*** fits into "example 3 vs 4" since both have similar outcome from an information perspective but only (3) seems related to physics. 

 

 

*) assuming a suitable definition exists
**) Or forgotten, it's a long time since I (briefly) studied thermodynamics.
***) feel free to post extra references; this is probably outside my current knowledge.
****) This would be a bad choice of implementation for this example in a real case, it's just used here to illustrate and compare reasons for loss of information. https://en.wikipedia.org/wiki/Lossy_compression

 

Edited March 16, 2022 by Ghideon
clarification & grammar

[studiot]

@Ghideon  and @joigus

Thank you both for a useful continuing scientific conversation.  +1  apiece.
There doesn't seem much else scientific going on here at the moment.

I think Shannon entropy is specified and calculated on the basis of 'ideal' computers devoid of earthly defects, including the need for power sources.

This is nothing new and continues the Ancient Greek tradition of abstracting perfect circles, squares etc as 'ideals'.

We have carried this tradition on thoughout history in both Philosophy, Engineering and more recently Physics.

In particular physical (ie thermodynamic) entropy and other thermodynamic properties are calculated on the basis of 'perfect' or ideal processes. Ghideon's comment about time applies if he doesn't already know this as the equations are almost all derived on the basis of infinitely slow (ideal) processes called reversible ones. thermodynamics doesn't care how long it takes to get there.

One difference is that the thermodynamic statement "Ideal entropy cannot decrease" is defined for a cyclic process.
It does not forbid entropic decrease within a cycle and this actually happens in some practical situations.
Computing processes are not, in general 'reversible' in the same way.

 

 

 

 

 

 

[Ghideon]

22 hours ago, studiot said:

In particular physical (ie thermodynamic) entropy and other thermodynamic properties are calculated on the basis of 'perfect' or ideal processes.

Thanks! This helps identifying where I need more reading / studying, I'm of course aware of perfect or ideal processes. My world view though is biased by working with software and models that can be assumed to be 'ideal' but are deployed in a 'non ideal' physical reality where computation and storage/retrieval/transmissions of (logical) information is affected by faulty components, neglected maintenance, lost documentation, power surges, bad decisions, miscommunications and what not. 

I think I should to approach this topic more in terms of ideal physics & thermodynamics. +1 for the helpful comment.

[joigus]

Thanks both for your useful comments. I'm starting to realise that the topic is vast, so fat chance that we can get to clarify all possible aspects.

I'm trying to get up to speed in the computer-science concept of entropy beyond the trivial difference of a scale factor that comes from logs in base e to logs in base 2. It may also interesting to notice that Shannon entropy is by no means only applied to messages, channels, senders, and receivers:

Quote

Data compression[edit]

Main articles: Shannon's source coding theorem and Data compression

Shannon's definition of entropy, when applied to an information source, can determine the minimum channel capacity required to reliably transmit the source as encoded binary digits. Shannon's entropy measures the information contained in a message as opposed to the portion of the message that is determined (or predictable).

(with my emphasis, and from https://en.wikipedia.org/wiki/Entropy_(information_theory)#Relationship_to_thermodynamic_entropy.)

In the meantime I've reminded myself as well that people are using entropy (whether base 2 or e) to discuss problems in evolution dynamics. See, e.g.,

https://en.wikipedia.org/wiki/Entropy_(information_theory)#Entropy_as_a_measure_of_diversity

https://www.youtube.com/watch?v=go6tC9VSME4&t=280s

Even in the realm of physics alone, there are quantum concepts, like entanglement entropy, quantum entropy a la Von Neumann, defined as a trace of matrices, or even entropy for pure states, \( -\int\left|\psi\right|^{2}\ln\left|\psi\right|^{2} \). The unifying aspect of all of them is that they can all be written one way or the other, schematically as,

Entropy(1,2,...,n)=-Sum(probability)log(probability)

for any partition into cells or states of your system that allow the definition of probabilities p(1), ..., p(n)

Entropy, quite simply, is a measure of how statistically-flattened-out whatever system you're dealing with, that admits a partition into identifyiable cells that play the part of "states", that is extensive (additive) for independent systems when considered together, and concave with respect to partitions into events:

Quote

• The entropy  is concave in the probability mass function , i.e.^[10]: 30

Being so general, it seems clear why it has far outstripped the narrow limits of its historical formulation.

Control parameters

Control parameters are any parameters that you can change at will. In the case of an ideal gas, typically we refer to control parameters as the extensive variables of the system, like the volume and the energy (those would be, I think, analogous to input data, program, ROM, because you can change them at will. Other things are going on within the computer that you cannot see. Result of a computation is unpredictable, as well as the running state of a program.

As to connection to physics, I suppose whether a particular circuit element is in a conducting state or interrupt one could be translated into physics.

But what's important to notice, I think, is that entropy can be used in many different ways that are not necessarily very easy to relate to the physical concept (the one that always must grow for the universe as a whole.)

As @studiot and myself have pointed out (I'm throwing in some other I'm not totally sure about):

• No analogue of thermodynamic equilibrium
• No cyclicity (related to ergodicity) => programs tend to occupy some circuits more likely than others 
• No reversibility
• No clear-cut concept of open vs closed systems (all computers are open)

But in the end, computers (I'm making no distinction with message-sending, so potentially I'm missing a lot here) are physical systems, so it would be possible in principle to write out their states as a series of conducting or non-conducting states subject to conservation of energy, and all the rest.

Please, @Ghideon, let me keep thinking about your distinctions (1)-(4), but for the time being, I think physics has somehow been factored out of the problem in (1) and (2), but is not completely not there, if you know what I mean. It's been made irrelevant as a consequence of the system having been idealised to the extreme. In cases (3) & (4) there would be no way to not make reference to it.

Going back to my analogy physics/computer science, it would go something like this:

Input (control parameters: V, E) --> data processing (equation of state f(V,P,T)=0) --> output (other parameters: P,T,)

Message coding would be similar, including message, public key, private key, coding function, etc. But it would make this post too long.

[Genady]

Since this thread has something to do with entropy - 😉 - I hope this question is not OT. Are you familiar with this book:

ENTROPY: The Greatest Blunder in the History of science

Arieh Ben-Naim

?

[joigus]

1 hour ago, Genady said:

Since this thread has something to do with entropy - 😉 - I hope this question is not OT. Are you familiar with this book:

ENTROPY: The Greatest Blunder in the History of science

Arieh Ben-Naim

?

I haven't heard about it. I've just read a short review and it sounds interesting. Thank you.

I would agree that entropy doesn't necessarily represent ignorance. Sometimes it's a measure of information contained in a system. An interesting distinction --that I'm not sure corresponds to professor Ben-Naim's criterion-- is between fine-grained entropy --which is to do with overall information content-- and coarse grained entropy --which is to do with available or controlled information.

OTOH, entropy is the concept which seems to be responsible for more people telling the rest of the world that nobody has understood it better than they have.  A distinguised example is Von Neumann*, and a gutter-level example is the unpalatable presence that we've had on this thread before you came to grace it with your contribution, Genady.

*https://mathoverflow.net/questions/403036/john-von-neumanns-remark-on-entropy

[studiot]

4 hours ago, Genady said:

Since this thread has something to do with entropy - 😉 - I hope this question is not OT. Are you familiar with this book:

ENTROPY: The Greatest Blunder in the History of science

Arieh Ben-Naim

?

Any chance of a glimpse of the contents page at least and a review would be nice.

I can't find these anywhere and I am unwilling to risk £20 to find out.

The author seems to have written several books about entropy, but I can only find a couple of sentences quoted.
These do seem to make sense to me however.

[Genady]

1 hour ago, studiot said:

Any chance of a glimpse of the contents page at least and a review would be nice.

I can't find these anywhere and I am unwilling to risk £20 to find out.

The author seems to have written several books about entropy, but I can only find a couple of sentences quoted.
These do seem to make sense to me however.

I've got it from Amazon US on Kindle for $9.99.

Here is a review from the Amazon page:

Quote

The Second Law of thermodynamics, the law of entropy, is one of the longest-standing laws of physics, unchanged even by the last century’s two drastic revolutions in physics.

However, the concept of entropy has long been misinterpreted and misused – making it the greatest ever blunder in the history of science, propagated for decades by scientists and non-scientists alike.

This blunder was initially and primarily brought on by a deep misunderstanding of the concept of entropy. Ironically, ignorance about the meaning of entropy has led some scientists to associate entropy with ignorance, and the Second Law with the “law of spreading ignorance.”

In his book, Arieh Ben-Naim, a respected professor of physical chemistry, attempts to right these wrongs. He scrutinizes twelve misguided definitions and interpretations of entropy, brings order to the chaos, and finally lays out the true meaning of entropy in clear and accessible language anyone can understand.

The contents:

Hope it helps. Glad to assist as much as I can.

[studiot]

Now that's a really good response. Thank you +1.

It seems serious stuff from the contents list,

quite at variance with the overdramatised brash headlines from Amazon.

I find myself with much more interest in reading this book.

Do you have any idea of the difference between this one and his others on the same subject ?

 

Meanwhile have you heard of this book ?

It has a similar theme and ambit.

 

Here is a very interesting timeline found in Appendix IV

 

 

Finally I don't know if you were here when I last stated the simple original reason for introducing the entropy function, which makes it so easy to explain.

Have you heard of indicator diagrams ?

 

[Genady]

I'm reading this book right now. Saw his other books on Amazon, but this one was the first I've picked. Interestingly, the Student's Guide book is the next on my list. I guess, you'd recommend it.

I didn't see when you stated the simple original reason for introducing the entropy function, and I haven't heard of indicator diagrams...

As you can see, this subject is relatively unfamiliar to me. I have had only a general idea from short intros to thermodynamics, statistical mechanics, and to information theory about 45 years ago. It is very interesting now, that I delve into it. Curiously, I've never needed it during long and successful career in computer systems design. So, not 'everything is information' even in that field, contrary to some members' opinion ...  

[studiot]

1 hour ago, Genady said:

I'm reading this book right now. Saw his other books on Amazon, but this one was the first I've picked. Interestingly, the Student's Guide book is the next on my list. I guess, you'd recommend it.

I didn't see when you stated the simple original reason for introducing the entropy function, and I haven't heard of indicator diagrams...

As you can see, this subject is relatively unfamiliar to me. I have had only a general idea from short intros to thermodynamics, statistical mechanics, and to information theory about 45 years ago. It is very interesting now, that I delve into it. Curiously, I've never needed it during long and successful career in computer systems design. So, not 'everything is information' even in that field, contrary to some members' opinion ...  

also @joigus

I have been looking round the net for information about Arieh Ben-naim and he seems to be a bit aside from the mainstream.

I downloaded this pdf of one of his aticles where he seems to spend a lot of effort attacking the works of others, rather than developing his own stuff, especially when it contains some rather suspect statements.

https://www.mdpi.com/1099-4300/21/12/1170

I would consider the book disappointing if that was all or the bulk of it's contents.

 

Back to the subject of entropy.

Ask yourself which group of people are most concerned with entropy ?

Answer those who have compiled very extensive tables of values and use them in their every day work.
Chemical Engineers, Mechanical Engineers and other Engineers directly concerned with Themodynamic processes.

The forerunners of these people were those who originally introduced and defined entropy for very practical purposes, a hundred years before Shannon.

One thing those steam engineers invented was called an indicator, which was a device to 'indicate' steam pressure.
This led to the birth of the 'indicator diagram which is a plot showing the energies involved in a particular process.
Originally these were P - V diagrams, the area under which indicates work.
It was desired to introduce a variable that coupled with temperature to indicate energy and that is why entropy was conceived and introduced and what you will find on modern day's entropy tables.
In modern times we have found quite a few pairs of variables that yield the dimensions of work or energy when multiplied together and also that it is useful to prepare 'indicator diagrams' for each of these.

 

 

[Genady]

Thank you for the explanation of indicator diagrams and of the introduction of thermodynamic entropy. Very clear! +1

I remember these diagrams from my old studies. They also show sometimes areas of different phases and critical points, IIRC. I didn't know, or didn't remember, that they are called indicator diagrams.

[Genady]

Regarding the Ben-Naim's book, unfortunately it in fact disappoints. I'm about 40% through and although it has clear explanations and examples, it goes for too long and too repetitiously into debunking of various metaphors, esp. in pop science. OTOH, these detours are easy to skip. The book could be shorter and could flow better without them.

[Genady]

However, while pointing to some others' mistakes he's clarified for me some subtle issues and I appreciate this.

[joigus]

15 hours ago, studiot said:

also @joigus

I have been looking round the net for information about Arieh Ben-naim and he seems to be a bit aside from the mainstream.

I downloaded this pdf of one of his aticles where he seems to spend a lot of effort attacking the works of others, rather than developing his own stuff, especially when it contains some rather suspect statements.

https://www.mdpi.com/1099-4300/21/12/1170

I would consider the book disappointing if that was all or the bulk of it's contents.

 

Thanks, @studiot. From skimming through the article I get the impression that prof. Beh-Naim goes a little bit over the top. He must be a hard-line Platonist. I'm rather content with accepting that whenever we have a system made up of a set of states, and we can define probabilities, frecuence of occupation, or the like for these states; then we can meaningfully define an entropy. What you're supposed to do with that function is another matter and largely depends on the nature of the particular system. In physics there are constraints that have to do with conservation principles, which winds up giving us the Maxwell-Boltzmann distribution. But it doesn't bother me that it's used in other contexts, even in physics. I'll read it more carefully later.

[Genady]

I looked in Steven Weinberg's book, Foundations of Modern Physics, for his way of introducing the thermodynamics' and the statistical mechanics' definitions of entropy and their connection, and I've noticed what seems to be an inconsistency. Here is how it goes:

Step 1. Quantity H = ∫ P(α) dα ln P(α) is defined for a system that can be in states parametrized by α, with probability distribution P(α). (This H is obviously equal negative Shannon's information entropy up to a constant, log₂e.)

Step 2. Shown that starting from any probability distribution, a system evolves in such a way that H of its probability distribution decreases until it reaches a minimal value.

Step 3. Shown that the probability distribution achieved at the end of step 2, with the minimal H, is a probability distribution at equilibrium.

Step 4. Shown that the minimal value of H achieved at the end of step 2, is proportional to negative entropy, S=-kH.

Step 5. Concluded that the decrease of H occurring on Step 2 implies the increase in  entropy.

The last step seems to be inconsistent, because only the minimal value of H achieved at the end of step 2 is shown to be  connected to entropy, and not the decreasing values of H before it reaches the minimum. Thus the decreasing H prior to that point cannot imply anything about entropy.

I understand that entropy might change as a system goes from one equilibrium state to another. This means that a minimal value of H achieved at one equilibrium state differs from a minimal value of H achieved at another equilibrium state. But this difference of minimal values is not the same as decreasing H from a non-minimum to a minimum.

[joigus]

11 hours ago, Genady said:

I looked in Steven Weinberg's book, Foundations of Modern Physics, for his way of introducing the thermodynamics' and the statistical mechanics' definitions of entropy and their connection, and I've noticed what seems to be an inconsistency. Here is how it goes:

Step 1. Quantity H = ∫ P(α) dα ln P(α) is defined for a system that can be in states parametrized by α, with probability distribution P(α). (This H is obviously equal negative Shannon's information entropy up to a constant, log₂e.)

Step 2. Shown that starting from any probability distribution, a system evolves in such a way that H of its probability distribution decreases until it reaches a minimal value.

Step 3. Shown that the probability distribution achieved at the end of step 2, with the minimal H, is a probability distribution at equilibrium.

Step 4. Shown that the minimal value of H achieved at the end of step 2, is proportional to negative entropy, S=-kH.

Step 5. Concluded that the decrease of H occurring on Step 2 implies the increase in  entropy.

The last step seems to be inconsistent, because only the minimal value of H achieved at the end of step 2 is shown to be  connected to entropy, and not the decreasing values of H before it reaches the minimum. Thus the decreasing H prior to that point cannot imply anything about entropy.

I understand that entropy might change as a system goes from one equilibrium state to another. This means that a minimal value of H achieved at one equilibrium state differs from a minimal value of H achieved at another equilibrium state. But this difference of minimal values is not the same as decreasing H from a non-minimum to a minimum.

OK, I've been thinking about this one for quite a while and I can't make up my mind whether you're just overthinking this or pointing out something deep.

I'm not sure this is what's bothering you, but for the \( P\left( \alpha \right) \) to be well-defined probabilities they should be some fixed frequency ratios to be checked in an infinitely-many-times reproducible experiment. So this \( -\int P\left(\alpha\right)\ln P\left(\alpha\right) \) already has to be the sought-for entropy, and not some kind of weird thing \( -\int P\left(\alpha,t\right)\ln P\left(\alpha,t\right) \).

IOW, how can you even have anything like time-dependent probabilities?

I'm not sure that's what you mean, but it touches on something that I've been thinking for a long time. Namely: that many fundamental ideas that we use in our theories are tautological to a great extent in the beginning, and they start producing results only when they're complemented with other ancillary hypotheses.

In the case of the second principle "derived" from pure mathematical reasoning, I think it's when we relate it to energy and number of particles, etc., and derive the Maxwell-Boltzmann distribution, that we're really in business. 

[Genady]

I don't know how deep it is, but I doubt that I'm overthinking this. I just follow the Weinberg's derivation of the H-theorem, which in turn follows the Gibbs' one, which in turn is a generalization of the Boltzmann's one.

The first part of this theorem is that dH/dt ≤ 0. That is, H is time-dependent, which means P(α) is time-dependent.

I thing that in order for H to correspond to a thermodynamic entropy, P(α) has to correspond to a well-defined thermodynamic state. However, in other states it actually is "some kind of weird thing −∫P(α,t)lnP(α,t)."

9 hours ago, joigus said:

−∫P(α,t)lnP(α,t) .how can you even have anything like time-dependent probabilities?

This is how I think it's done, for example. Let's consider an isolated box of volume 2V with a partition in the middle. The left half, of volume V has one mole of an ideal gas inside with a total energy E. The right half is empty. Now let's imagine that we have infinitely-many of these boxes. At some moment, let's remove the partitions in all of them at once and then let's take snapshots of the states of gas in all the boxes at a fixed time t. We'll get a distribution of states of gas in the process of filling the boxes, and this distribution depends on t. This distribution gives us the P(α,t) . 

The H-theorem says that, as these distributions change in time the corresponding H decreases until it settles at a minimum value when a distribution corresponds to a well-defined thermodynamic state. At that state it is the same as -kS.   

[studiot]

3 hours ago, Genady said:

This is how I think it's done, for example. Let's consider an isolated box of volume 2V with a partition in the middle. The left half, of volume V has one mole of an ideal gas inside with a total energy E. The right half is empty. Now let's imagine that we have infinitely-many of these boxes. At some moment, let's remove the partitions in all of them at once and then let's take snapshots of the states of gas in all the boxes at a fixed time t. We'll get a distribution of states of gas in the process of filling the boxes, and this distribution depends on t. This distribution gives us the P(α,t) . 

+1 for the example to discuss.

Not sure I see time in this, but this thought experiment on an ideal system introduces two interesting facts.

Firstly in terms of classical Thermodynamics, the general equation for the entropy chance of such an expansion can be shown to be

[math]\Delta S = {C_v}\ln \left( {\frac{{{T_2}}}{{{T_1}}}} \right) + nR\ln \left( {\frac{{{V_2}}}{{{V_1}}}} \right) = {C_v}\ln \left( {\frac{{{T_2}}}{{{T_1}}}} \right) + nR\ln \left( {\frac{{{P_1}}}{{{P_2}}}} \right)[/math]

In this case since q = w = 0 no work is done and no heat is exchanged with the surroundings

Therefore  [math]\Delta U = 0[/math], and thus for an ideal gas T₂ = T₁

Yet there is a non zero entropy change, due to the volume change alone since the integral of q/T must be zero.

Thius there is more to entropy itself than just the integral of q/T .

 

Comparing this to a digital computer system let us say we have such an ideal computer and the analog of volume would be memory capacity.
If we suddenly doubled the memory capacity, would there be a corresponding Shannon Entropy increase, since we have added nothing to the contents of that memory by the expansion ?

And how does information fit into both these scenarios ?

 

[Genady]

3 hours ago, studiot said:

Comparing this to a digital computer system let us say we have such an ideal computer and the analog of volume would be memory capacity.
If we suddenly doubled the memory capacity, would there be a corresponding Shannon Entropy increase, since we have added nothing to the contents of that memory by the expansion ?

And how does information fit into both these scenarios ?

IMO, the answer to the first question depends on conditions of usage of the memory. In the extreme case when the memory is added but the system is not using it, there is no entropy increase.

Regarding the second question, a probability distribution of the memory contents has information entropy. A probability distribution of micro-states of the gas in the box has information entropy as well. This is my understanding. It is interesting to run various scenarios and see how it works.