Boot entries keep accumulating / ThinkPad X220 does not wake up from suspend-to-ram

I have a ThinkPad X220 Tablet which runs Fedora 23 with automatic updates,
Chromium and Spotify Copr. Up until Friday night, 2016-03-19, it worked just
fine.

Friday afternoon the new Kernel 4.4.5 was installed. It was only loaded
on Saturday when I started the machine in the afternoon. I suspended the
laptop to RAM and found that it did not woke up. I only got this strange
light show:

https://www.youtube.com/watch?v=ajoWfCtk7yE

The error (sadly) is 100% reproducible. I then went on and started with the
earlier kernels, 4.4.4 and 4.4.3, which worked fine before Friday. The error is
just the same, it does not wake up.

Then I also started a Kubuntu 15.04 from USB and had a slightly
different error. The machine would start the fan and the power light
would turn on, but the wifi was still off and the screen was black.

Using Arch Linux 2016.03.01 I was able to reproduce the error with
Kernel 4.4.1. Therefore I think it is not about the software any more.
This might be somewhere in the hardware or UEFI firmware.

On the fedora-devel mailinglist I posted the above and got some valuable
feedback. In the meantime I noticed that I could not save anything in the UEFI.
It looks like this:

I tried to do an UEFI firmware upgrade using the boot CD images that Lenovo
provides. They are 16-bit DOS and my boot sequence was locked into UEFI only.
Therefore I had to boot a Windows and do the upgrade there. This did not change
much, I did not get any error message in the UEFI/BIOS but it would just freeze
when trying to save something.

Chris Murphy from the fedora-devel list suggested to look at the output of
efibootmgr -v. Currently it looks like the following:

BootCurrent: 000A
Timeout: 0 seconds
BootOrder: 004A,000A,0009,0006,0007,0008,000B,000C,000D,000E,000F,0010,0011,0012,0013
Boot0000  Setup FvFile(721c8b66-426c-4e86-8e99-3457c46ab0b9)
Boot0001  Boot Menu     FvFile(126a762d-5758-4fca-8531-201a7f57f850)
Boot0002  Diagnostic Splash Screen      FvFile(a7d8d9a6-6ab0-4aeb-ad9d-163e59a7a380)
Boot0003  Startup Interrupt Menu        FvFile(f46ee6f4-4785-43a3-923d-7f786c3c8479)
Boot0004  ME Configuration Menu FvFile(82988420-7467-4490-9059-feb448dd1963)
Boot0005  Rescue and Recovery   FvFile(665d3f60-ad3e-4cad-8e26-db46eee9f1b5)
Boot0006* USB CD        VenMsg(bc7838d2-0f82-4d60-8316-c068ee79d25b,86701296aa5a7848b66cd49dd3ba6a55)
Boot0007* USB FDD       VenMsg(bc7838d2-0f82-4d60-8316-c068ee79d25b,6ff015a28830b543a8b8641009461e49)
Boot0008* ATAPI CD0     VenMsg(bc7838d2-0f82-4d60-8316-c068ee79d25b,aea2090adfde214e8b3a5e471856a35401)
Boot0009* ATA HDD2      VenMsg(bc7838d2-0f82-4d60-8316-c068ee79d25b,91af625956449f41a7b91f4f892ab0f602)
Boot000A* ATA HDD0      VenMsg(bc7838d2-0f82-4d60-8316-c068ee79d25b,91af625956449f41a7b91f4f892ab0f600)
Boot000B* ATA HDD1      VenMsg(bc7838d2-0f82-4d60-8316-c068ee79d25b,91af625956449f41a7b91f4f892ab0f601)                
Boot000C* USB HDD       VenMsg(bc7838d2-0f82-4d60-8316-c068ee79d25b,33e821aaaf33bc4789bd419f88c50803)                  
Boot000D* PCI LAN       VenMsg(bc7838d2-0f82-4d60-8316-c068ee79d25b,78a84aaf2b2afc4ea79cf5cc8f3d3803)                  
Boot000E* ATAPI CD1     VenMsg(bc7838d2-0f82-4d60-8316-c068ee79d25b,aea2090adfde214e8b3a5e471856a35403)                
Boot000F* ATAPI CD2     VenMsg(bc7838d2-0f82-4d60-8316-c068ee79d25b,aea2090adfde214e8b3a5e471856a35404)                
Boot0010  Other CD      VenMsg(bc7838d2-0f82-4d60-8316-c068ee79d25b,aea2090adfde214e8b3a5e471856a35406)                
Boot0011* ATA HDD3      VenMsg(bc7838d2-0f82-4d60-8316-c068ee79d25b,91af625956449f41a7b91f4f892ab0f603)                
Boot0012* ATA HDD4      VenMsg(bc7838d2-0f82-4d60-8316-c068ee79d25b,91af625956449f41a7b91f4f892ab0f604)                
Boot0013  Other HDD     VenMsg(bc7838d2-0f82-4d60-8316-c068ee79d25b,91af625956449f41a7b91f4f892ab0f606)                
Boot0014* IDER BOOT CDROM       PciRoot(0x0)/Pci(0x16,0x2)/Ata(0,1,0)                                                  
Boot0015* IDER BOOT Floppy      PciRoot(0x0)/Pci(0x16,0x2)/Ata(0,0,0)                                                  
Boot0016* ATA HDD       VenMsg(bc7838d2-0f82-4d60-8316-c068ee79d25b,91af625956449f41a7b91f4f892ab0f6)                  
Boot0017* ATAPI CD:     VenMsg(bc7838d2-0f82-4d60-8316-c068ee79d25b,aea2090adfde214e8b3a5e471856a354)                  
Boot0018* PCI LAN       VenMsg(bc7838d2-0f82-4d60-8316-c068ee79d25b,78a84aaf2b2afc4ea79cf5cc8f3d3803)                  
Boot0019* kubuntu       HD(1,GPT,9feb65d0-1f99-4f66-8cc1-1e2aa3c71354,0x800,0xf3800)/File(\EFI\kubuntu\shimx64.efi)    
Boot001A* Fedora        HD(1,GPT,18add345-44b0-44dd-9aa9-feae671b7d2e,0x800,0x64000)/File(\EFI\fedora\shim.efi)        
Boot001B* Fedora        HD(1,GPT,18add345-44b0-44dd-9aa9-feae671b7d2e,0x800,0x64000)/File(\EFI\fedora\shim.efi)        
Boot001C* Fedora        HD(1,GPT,18add345-44b0-44dd-9aa9-feae671b7d2e,0x800,0x64000)/File(\EFI\fedora\shim.efi)        
Boot001D* Fedora        HD(1,GPT,18add345-44b0-44dd-9aa9-feae671b7d2e,0x800,0x64000)/File(\EFI\fedora\shim.efi)        
Boot001E* Fedora        HD(1,GPT,18add345-44b0-44dd-9aa9-feae671b7d2e,0x800,0x64000)/File(\EFI\fedora\shim.efi)        
Boot001F* Fedora        HD(1,GPT,18add345-44b0-44dd-9aa9-feae671b7d2e,0x800,0x64000)/File(\EFI\fedora\shim.efi)        
Boot0020* Fedora        HD(1,GPT,18add345-44b0-44dd-9aa9-feae671b7d2e,0x800,0x64000)/File(\EFI\fedora\shim.efi)        
Boot0021* Fedora        HD(1,GPT,18add345-44b0-44dd-9aa9-feae671b7d2e,0x800,0x64000)/File(\EFI\fedora\shim.efi)        
Boot0022* Fedora        HD(1,GPT,18add345-44b0-44dd-9aa9-feae671b7d2e,0x800,0x64000)/File(\EFI\fedora\shim.efi)        
Boot0023* Fedora        HD(1,GPT,18add345-44b0-44dd-9aa9-feae671b7d2e,0x800,0x64000)/File(\EFI\fedora\shim.efi)        
Boot0024* Fedora        HD(1,GPT,18add345-44b0-44dd-9aa9-feae671b7d2e,0x800,0x64000)/File(\EFI\fedora\shim.efi)        
Boot0025* Fedora        HD(1,GPT,18add345-44b0-44dd-9aa9-feae671b7d2e,0x800,0x64000)/File(\EFI\fedora\shim.efi)        
Boot0026* Fedora        HD(1,GPT,18add345-44b0-44dd-9aa9-feae671b7d2e,0x800,0x64000)/File(\EFI\fedora\shim.efi)        
Boot0027* Fedora        HD(1,GPT,18add345-44b0-44dd-9aa9-feae671b7d2e,0x800,0x64000)/File(\EFI\fedora\shim.efi)        
Boot0028* Fedora        HD(1,GPT,18add345-44b0-44dd-9aa9-feae671b7d2e,0x800,0x64000)/File(\EFI\fedora\shim.efi)        
Boot0029* Fedora        HD(1,GPT,18add345-44b0-44dd-9aa9-feae671b7d2e,0x800,0x64000)/File(\EFI\fedora\shim.efi)
Boot002A* Fedora        HD(1,GPT,18add345-44b0-44dd-9aa9-feae671b7d2e,0x800,0x64000)/File(\EFI\fedora\shim.efi)
Boot002B* Fedora        HD(1,GPT,18add345-44b0-44dd-9aa9-feae671b7d2e,0x800,0x64000)/File(\EFI\fedora\shim.efi)
Boot002C* Fedora        HD(1,GPT,18add345-44b0-44dd-9aa9-feae671b7d2e,0x800,0x64000)/File(\EFI\fedora\shim.efi)
Boot002D* Fedora        HD(1,GPT,18add345-44b0-44dd-9aa9-feae671b7d2e,0x800,0x64000)/File(\EFI\fedora\shim.efi)
Boot002E* Fedora        HD(1,GPT,18add345-44b0-44dd-9aa9-feae671b7d2e,0x800,0x64000)/File(\EFI\fedora\shim.efi)
Boot002F* Fedora        HD(1,GPT,18add345-44b0-44dd-9aa9-feae671b7d2e,0x800,0x64000)/File(\EFI\fedora\shim.efi)
Boot0030* Fedora        HD(1,GPT,18add345-44b0-44dd-9aa9-feae671b7d2e,0x800,0x64000)/File(\EFI\fedora\shim.efi)
Boot0031* Fedora        HD(1,GPT,18add345-44b0-44dd-9aa9-feae671b7d2e,0x800,0x64000)/File(\EFI\fedora\shim.efi)
Boot0032* Fedora        HD(1,GPT,18add345-44b0-44dd-9aa9-feae671b7d2e,0x800,0x64000)/File(\EFI\fedora\shim.efi)
Boot0033* Fedora        HD(1,GPT,18add345-44b0-44dd-9aa9-feae671b7d2e,0x800,0x64000)/File(\EFI\fedora\shim.efi)
Boot0034* Fedora        HD(1,GPT,18add345-44b0-44dd-9aa9-feae671b7d2e,0x800,0x64000)/File(\EFI\fedora\shim.efi)
Boot0035* Fedora        HD(1,GPT,18add345-44b0-44dd-9aa9-feae671b7d2e,0x800,0x64000)/File(\EFI\fedora\shim.efi)
Boot0036* Fedora        HD(1,GPT,18add345-44b0-44dd-9aa9-feae671b7d2e,0x800,0x64000)/File(\EFI\fedora\shim.efi)
Boot0037* Fedora        HD(1,GPT,18add345-44b0-44dd-9aa9-feae671b7d2e,0x800,0x64000)/File(\EFI\fedora\shim.efi)
Boot0038* Fedora        HD(1,GPT,18add345-44b0-44dd-9aa9-feae671b7d2e,0x800,0x64000)/File(\EFI\fedora\shim.efi)
Boot0039* Fedora        HD(1,GPT,18add345-44b0-44dd-9aa9-feae671b7d2e,0x800,0x64000)/File(\EFI\fedora\shim.efi)
Boot003A* Fedora        HD(1,GPT,18add345-44b0-44dd-9aa9-feae671b7d2e,0x800,0x64000)/File(\EFI\fedora\shim.efi)
Boot003B* Fedora        HD(1,GPT,18add345-44b0-44dd-9aa9-feae671b7d2e,0x800,0x64000)/File(\EFI\fedora\shim.efi)
Boot003C* Fedora        HD(1,GPT,18add345-44b0-44dd-9aa9-feae671b7d2e,0x800,0x64000)/File(\EFI\fedora\shim.efi)
Boot003D* Fedora        HD(1,GPT,18add345-44b0-44dd-9aa9-feae671b7d2e,0x800,0x64000)/File(\EFI\fedora\shim.efi)
Boot003E* Fedora        HD(1,GPT,18add345-44b0-44dd-9aa9-feae671b7d2e,0x800,0x64000)/File(\EFI\fedora\shim.efi)
Boot003F* Fedora        HD(1,GPT,18add345-44b0-44dd-9aa9-feae671b7d2e,0x800,0x64000)/File(\EFI\fedora\shim.efi)
Boot0040* Fedora        HD(1,GPT,18add345-44b0-44dd-9aa9-feae671b7d2e,0x800,0x64000)/File(\EFI\fedora\shim.efi)
Boot0041* Fedora        HD(1,GPT,18add345-44b0-44dd-9aa9-feae671b7d2e,0x800,0x64000)/File(\EFI\fedora\shim.efi)
Boot0042* Fedora        HD(1,GPT,18add345-44b0-44dd-9aa9-feae671b7d2e,0x800,0x64000)/File(\EFI\fedora\shim.efi)
Boot0043* Fedora        HD(1,GPT,18add345-44b0-44dd-9aa9-feae671b7d2e,0x800,0x64000)/File(\EFI\fedora\shim.efi)
Boot0044* Fedora        HD(1,GPT,18add345-44b0-44dd-9aa9-feae671b7d2e,0x800,0x64000)/File(\EFI\fedora\shim.efi)
Boot0045* Fedora        HD(1,GPT,18add345-44b0-44dd-9aa9-feae671b7d2e,0x800,0x64000)/File(\EFI\fedora\shim.efi)
Boot0046* Fedora        HD(1,GPT,18add345-44b0-44dd-9aa9-feae671b7d2e,0x800,0x64000)/File(\EFI\fedora\shim.efi)
Boot0047* Fedora        HD(1,GPT,18add345-44b0-44dd-9aa9-feae671b7d2e,0x800,0x64000)/File(\EFI\fedora\shim.efi)
Boot0048* Fedora        HD(1,GPT,18add345-44b0-44dd-9aa9-feae671b7d2e,0x800,0x64000)/File(\EFI\fedora\shim.efi)
Boot0049* Fedora        HD(1,GPT,18add345-44b0-44dd-9aa9-feae671b7d2e,0x800,0x64000)/File(\EFI\fedora\shim.efi)
Boot004A* Fedora        HD(1,GPT,18add345-44b0-44dd-9aa9-feae671b7d2e,0x800,0x64000)/File(\EFI\fedora\shim.efi)

By his suggestion, I deleted a lot of the boot entries. He said that this might
be NVRAM corruption. The first delete that I tried using efibootmgr failed as
it complained about lack of free memory. So some hard limit has been reached
there. Deleing it via the efivars file system helped, and then I could delete
the remainder with efibootmgr.

He further suggested to delete everything in EFI/BOOT/ and run dnf reinstall shim grub2-efi, which I did as well as grub2-mkconfig -o /boot/efi/EFI/fedora/grub.cfg. After that, the checksum of
/boot/efi/EFI/BOOT/BOOTX64.EFI has not changed, though.

After I deleted the boot entries, I could make changes in the UEFI/BIOS screen
again. Also the laptop consistently wakes up from the suspend again.

As a workaround, I wrote a little script that deletes the Fedora boot entries
once there are over fifty.

In case it is of any interest, this is the output of lsblk:

NAME                                          MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINT
sda                                             8:0    0 465,8G  0 disk  
├─sda1                                          8:1    0   200M  0 part  /boot/efi
├─sda2                                          8:2    0   500M  0 part  /boot
└─sda3                                          8:3    0 465,1G  0 part  
  └─luks-3f530000-b2c3-4ba3-9e85-1a96494cc25d 253:0    0 465,1G  0 crypt 
    ├─fedora_martin--friese-root              253:1    0    50G  0 lvm   /
    ├─fedora_martin--friese-swap              253:2    0   7,8G  0 lvm   [SWAP]
    └─fedora_martin--friese-home              253:3    0 407,3G  0 lvm   /home

I am not sure whether this is the right place to report this bug. Is this
something which could be fixed withing this project?

No source file or test is named. Start by determining whether shim is implicated or whether the failure is confined to UEFI/NVRAM, using the reported efibootmgr output and the Fedora shim/grub reinstall steps; done would require a reproducible shim-level failure and a scoped project fix.