# Discuss ch 1 - 2.6 of "Elements of Information Theory" [series meeting]

• December 18, 2013 · 7:00 PM

Here's a popular text in Information Theory which goes into some detail on data compression and communication channels. Definitely a good one to read through!
http://www.amazon.com/Elements-Information-Theory-Telecommunications-Processing/dp/0471062596

For the first meet up, we'll get together and go over the first chapter (an introduction/overview) and then get into the fun stuff!  We'll stop after section 2.6 on Jensen's Inequality.  We'll work some problems together, and help everyone understand the material before moving on to the rest of the chapter!  I'm currently working on the venue, and can likely host it at work.

After 5 or more people comment saying they've read the material, we'll set a date and location!

Recommended Prerequisites:

Probability theory: (moderate level) Joint distributions and marginalization; conditioning; Bayes' Theorem

• Analysis: (beginner to moderate level) Know what a set is; know what a logarithm is; understand the basic ideas of proofs by induction, contradiction, etc.

• ##### Kirill

Thank you, Adam, Andrew for hosting a great meetup.

December 21

Hey Everyone! That was a good first meeting! Just for reference, we decided:
1) We'll do a book meeting every other week, and a paper meeting in between (every other week). That keeps up the pace of learning, while not going too fast through the books for people with busier schedules.
2) We'll pick problems for the book meeting (I'll probably post some recommended ones) and volunteer via the message board to work out/present the problems to the group. If nobody volunteers, I'll have a couple [pre-solved problems :) ] ready to present myself.

I think it goes without saying that these are just guidelines, and we're always open for some Q&A/group discussion. Problems are a great way to bring out the finer aspects of the material, and we can always break the presentation format to clarify the material.

December 19

Thomas: yep! We decided to read the rest of Chapter 2. We're going to meet again the second Wednesday of January (the first is New Year's Day).

December 19

Ivan: We didn't compare the two intentionally, but there's a possibility one is a little thicker than the other. We'll have to compare next time. In the meantime, I'll post the subheadings of each chapter in the details for the next meet up, so you can make sure all the content is there.

December 19

• ##### John Mangual

RSVP is closed. Can I still show up?

December 18

• ##### Aneesh Devi

Novice just getting interested in information theory

December 16

I've filled out the form to request a room at my work (Buzzfeed) for 6:30 Wednesday the 18th. I'll set the meetup date/time once I've heard confirmation.

December 9

noooooo! Ok. hahaha. next time? after the holidays?

December 12

• ##### Mike Selender

I have another Meetup for this date, but that event is on the same block, so maybe I'll be able to stop in for a while.

December 13

Hey Everyone! Just wanted to say around this time tomorrow is the cutoff for when I'll be tallying all the preferred meeting days, and reserving the venue! Please be sure to let me know when the week after next is the best day for you to meet!

December 4

• ##### Thomas Nyberg

Just piping in to say that I've done the reading.

December 1

(and in the future, we'll definitely choose books more democratically)

December 1

• ##### Thomas Nyberg

I assume that the material in these sections is pretty fundamental and probably found in almost every book on this subject. So I don't think that there would necessarily be much lost if we were to switch. That said, I don't think it's really that important to have the perfect book for the topic anyway...putting effort into any (reasonably chosen) book on the subject is much better than fussing about which one is best. Presumably the discussion of the material would flush out any ideas/opinions that might be covered different in other books. First it's good to get moving in any direction...then you worry about whether you need to turn the wheel.

2 · December 1

• ##### archisman rudra

I read it early in my grad school. The algebraic aspects are well covered. At least, you will get a nodding acquaintance with connections with coding theory, and the relationship between bandwidth, quantization accuracy and so on. The analytical side of the book is weak. I believe(it's been a while) only weak convergence is covered, which leads to much weaker results. This is easily remedied by supplementing with something like Kinchin's book, which is available Dover edition, as well as with Shannon's book/paper, which also seems to be carried by Amazon. One thing I liked about the Cover and Thomas book is the coverage of the typical sequence approach. Application to statistics is also covered, but is a little cursory, but there are definitely other books that can serve for deeper studies.

November 28

• ##### archisman rudra

Kirill, I did not want to suggest it is a bad book at all. I think it is a great first book. The first few chapters are probably all that is needed for this. Adam, I will try to get some of that statistics from information theory out. It has been some time. There is of course, the information geometry stuff also, and that is related, but that easy

1 · December 1

• ##### archisman rudra

Sorry, my typing clearly needs work. I seem to hit enter instead of backspace. Anyway, I will try to fig up the statistics reference.

December 1

• ##### ENGRPOD

I CAN ONLY GO TO THE DISCUSSION ON INFO THEORY IF IT'S REASONABLY NEARBY. IT WOULD NEED TO BE SOMEWHERE IN NASSAU, OR WESTERN SUFFOLK, NOWHERE IN NYC, THERE, I CANNOT GET TO, UNFORTUNATELY. ONCE WHERE IT WOULD BE MIGHT BE KNOWN, PLEASE TRY TO SAY WHERE, AND WHEN, IF AT ALL POSSIBLE, OBVIOUSLY, MANY THANKS.

November 27

Hi Engrpod! It's almost certainly going to be in the city, but I'll keep you posted on whether we'll be able to ustream it or something similar. may be a little better than skype.

I'm quite familiar with arxiv.org! we'll almost certainly read some things from it from time to time.

November 28

• ##### Serge

Chapter 1 is titled Introduction and Preview, and it consists of a single section, Preview of The Book, which is 8 pages long.

November 27

I suppose that won't give us many problems to solve! My mistake: we should do at least the first part of chapter 2 as well. Let's go up through Jensen's Inequality, sec 2.6 in the second edition.

November 27

• ##### Mike Selender

Is this the first or second edition?

November 27

you can compare your table of contents with the link above to make sure you have the same chapters. I'd bet some of the problems are different … we can probably work out a photocopying thing if you need copies.

November 27

I have the second edition, but I'd wager they're similar enough. Here's the second edition:
http://www.amazon.com/Elements-...­

November 27

• ##### Ivan Grosny

Hi Adam, here is a nice list of programming books http://cspray.github.io/my.so-a...­

And here a great list of free books on Algorithms
http://resrc.io/list/10/list-of...­

November 27

Thanks, Ivan! I'll start a page where we can all share resources!

November 27

November 27

• ##### ENGRPOD

ALSO, IF GETTING TO THE DISCUSSIONS WOULD BE POSSIBLE FOR ME, I DO HAVE SEVERAL OTHER TOPICS OF POSSIBLE INTEREST FOR ALL OF YOU AS WELL, USING CURRENT PAPERS FOUND ON WWW/ARXIV.ORG, IF ANY OF YOU KNOW THAT SITE. JUST ALSO SO ALL OF YOU WOULD KNOW, THERE IS A GROUP THAT GOES BY THE NAME NYC PHYSICS AND MATH AUTODIDACTS ON MEETUP.COM, THAT MEET, I GATHER, AT ONE OF THE MAJOR LIBRARIES IN BROOKLYN. I ONCE TRIED TO LISTEN INTO THEM THROUGH SKYPE, THOUGH THE CONNECTION WAS PRETTY BAD, UNFORTUNATELY. HOWEVER, I ONLY MENTION IT SINCE, GIVEN THE APPARENT INTEREST IN MATERIAL OF COMPARABLE CALIBER TO THEIRS, I THOUGHT ALL OF YOU MIGHT CARE TO POSSIBLY LOOK INTO MAYBE SEEKING TO AFFILIATE WITH THEM. FROM WHAT E-MAILS FROM THEM I'VE GOTTEN, THEY SEEM TO BE PRETTY ENGROSSED IN CLASSICAL MECHANICS, USING THE STANDARD TEXT BY GOLDSTEIN. THEY MAY HAVE OTHER INTERESTS, HOWEVER, THAT IS THE ONE I MAINLY HAVE READ THEY HAVE BEEN GOING INTO OF LATE, AND I GATHER THEY EVIDENTLY SEEM TO BE FAIRLY ACTIVE.

November 27

Once 5 (or more) people comment that they've read the first chapter, we'll set a location and date to discuss! We'll also set a schedule for going through the rest of the book.

November 27

### 19 went

• ##### andy huang
• A former member
• A former member
• A former member

### New York, NY

Founded Nov 21, 2013

Contact

#### People in this Meetup are also in:

• ##### NY Tech Meetup

40,447 NYC Technologists

Next Meetup:
Monday, Aug 04

• ##### NYC-GDG

3,127 members

Next Meetup:
Tuesday, Aug 26

• ##### BIG DATA PALOOZA

371 BIG DATA ACTIVISTS

• ##### NYLUG - New York Linux Users Group

2,672 Members

Next Meetup:
Wednesday, Aug 20

• ##### SQL NYC, The NoSQL & NewSQL Database Meetup

6,463 Data Enthusiasts

Next Meetup:
Monday, Sep 22